Dynamic tiling for foveated rendering

ABSTRACT

A method for providing imagery to a user on a display includes receiving eye tracking data and determining a gaze location on the display or a gaze vector using the eye tracking data. The method can also include defining a first tile using the gaze location on the display or the gaze vector. The first tile may have a height and a width, the height and width being determined using the eye tracking data. The method can further include defining multiple additional tiles to fill an entire area of the display. The method can also include providing a portion of an image using the first tile at a first image quality and providing another portion of the image at a second image quality using at least one of the multiple additional tiles.

CROSS-REFERENCE TO RELATED APPLICATION

The present application claims the benefit of and priority to U.S.Provisional Patent No. 62/861,106, filed Jun. 13, 2019, the entiredisclosure of which is incorporated by reference herein.

FIELD OF THE DISCLOSURE

The present disclosure relates generally to display systems. Moreparticularly, the present disclosure relates to systems and methods forusing eye tracking with foveated rendering.

BACKGROUND

The present disclosure relates generally to augmented reality (AR),mixed reality (MR) and/or virtual reality (VR) systems. AR, MR, and VRsystems can be used to present various images, includingthree-dimensional (3D) images, to a user. For example, AR, MR or VRheadsets can be used to present images to the user in a manner that isoverlaid on a view of a real world environment or that simulates avirtual environment. To render convincing, life-like AR/MR/VR images,the AR/MR/VR systems can use eye tracking to track the user's eye andaccordingly present images.

SUMMARY

One implementation of the present disclosure is a method for providingimagery to a user on a display. The method may include receiving eyetracking data. The method may further include determining a gazelocation on the display or a gaze vector using the eye tracking data.The method can further include defining a first tile using the gazelocation on the display or the gaze vector. The first tile may have aheight and a width, the height and width being determined using the eyetracking data. The method can further include defining multipleadditional tiles to fill an entire area of the display. The method canalso include providing a portion of an image using the first tile at afirst image quality and providing another portion of the image at asecond image quality using at least one of the multiple additionaltiles.

Another implementation of the present disclosure is a head mounteddisplay for providing imagery to a user. The head mounted display caninclude a display, and eye tracker, and a processor. The eye tracker canbe configured to provide eye tracking data. The processor can beconfigured to determine a gaze location on the display or a gaze vectorusing the eye tracking data. The processor can further be configured todefine a first tile using the gaze location or the gaze vector. Theprocessor can further be configured to define multiple additional tilesto fill an area of the display. A portion of an image may be provided onthe display using the first tile at a first image quality and otherportions of the image may be provided on the display at a second imagequality using at least one of the multiple additional tiles.

Another implementation of the present disclosure is a display forproviding foveated imagery to a user. The display may include processingcircuitry configured to track a gaze direction or a gaze vector of theuser's eye. The processing circuitry may be further configured to definea first tile based on the gaze direction or gaze vector of the user'seye. A location of the first tile is defined using the gaze direction orthe gaze vector and is for a first image quality. The processingcircuitry may be configured to define one or more additional tiles for asecond image quality different than the first image quality. Theprocessing circuitry can be configured to provide imagery using thefirst tile and each of the one or more additional tiles. The processingcircuitry may also be configured to redefine the location of the firsttile in response to a change in the gaze direction or gaze vector.

These and other aspects and implementations are discussed in detailbelow. The foregoing information and the following detailed descriptioninclude illustrative examples of various aspects and implementations,and provide an overview or framework for understanding the nature andcharacter of the claimed aspects and implementations. The drawingsprovide illustration and a further understanding of the various aspectsand implementations, and are incorporated in and constitute a part ofthis specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. Likereference numbers and designations in the various drawings indicate likeelements. For purposes of clarity, not every component can be labeled inevery drawing. In the drawings:

FIG. 1 is a block diagram of a display system, according to someembodiments.

FIG. 2 is a schematic diagram of a head-mounted display (HMD) system,according to some embodiments.

FIG. 3 is a spherical coordinate system showing a gaze vector of auser's eye, according to some embodiments.

FIG. 4 is a top view of the gaze vector of FIG. 3 directed towards adisplay screen, according to some embodiments.

FIG. 5 is a side view of the gaze vector of FIG. 3 directed towards adisplay screen, according to some embodiments.

FIG. 6 is a tiled display screen with foveated rendering, according tosome embodiments.

FIG. 7 is the tiled display screen of FIG. 6 with foveated renderingafter the user's gaze has shifted to a new location, according to someembodiments.

FIG. 8 is the tiled display screen of FIG. 6 with foveated rendering fora first gaze position error, according to some embodiments.

FIG. 9 is the tiled display screen of FIG. 6 with foveated rendering fora second gaze position error, according to some embodiments.

FIG. 10 is a tiled display screen with smaller tiles than the displayscreen of FIGS. 6-9 and foveated rendering, according to someembodiments.

FIG. 11 is the tiled display screen of FIG. 10 with foveated renderingafter a user's gaze has shifted to a new location, according to someembodiments.

FIG. 12 is a portion of a tiled display screen with a user's gazecentered at a first tile, according to some embodiments.

FIG. 13 is the portion of the tiled display screen of FIG. 12 after theuser's gaze has shifted to a new location near a tile border and a newtile has been defined, according to some embodiments.

FIG. 14 is the portion of the tiled display screen of FIG. 12 withsmaller gaze position error and smaller tiles, according to someembodiments.

FIG. 15 is the portion of the tiled display screen of FIG. 14 after theuser's gaze has shifted to a new location near a tile border and newtiles have been defined to account for the shifted user's gaze,according to some embodiments.

FIG. 16 is a flow diagram of a process for dynamically tiling a displayscreen and providing foveated rendering based on gaze direction,according to some embodiments.

FIG. 17 is a flow diagram of a process for dynamically defining one ormore new tiles based on a user's gaze position, according to someembodiments.

FIG. 18 is a flow diagram of a process for determining if new tilesshould be defined based on a user's gaze, according to some embodiments.

FIG. 19 is a flow diagram of a process for determining if new tilesshould be defined based on gaze error, according to some embodiments.

FIG. 20 is a block diagram of a computing environment that the systemsof FIGS. 1 and 2 can be implemented in, according to some embodiments.

FIG. 21 is a front view of a lens of a display with various imagequality regions, according to some embodiments.

DETAILED DESCRIPTION

Overview

Referring generally to the FIGURES, systems and methods for providingfoveated images to a user are shown, according to some embodiments.Tiles are used to display the foveated images, according to someembodiments. A user's eye is tracked to determine gaze direction and/orfocal point, according to some embodiments. The gaze direction and/orfocal point is used to determine a gaze location on a display, accordingto some embodiments. The gaze location can be a gaze location (X, Y) ona two dimensional display or a gaze location (X, Y, Z) on a threedimensional display. In some embodiments, an error associated with thegaze direction and/or the gaze location on the display is alsodetermined.

Various eye tracking sensors, devices, hardware, software, etc., areused to track the user's eye and to determine the gaze location on thedisplay, according to some embodiments. A tile is defined that iscentered at the gaze location on the display, and additional tiles arealso defined to fill out remaining area of the display, according tosome embodiments. The tile that is centered at the gaze location on thedisplay is updated in real-time to track the user's gaze direction as itchanges, according to some embodiments. In some embodiments, the tilethat is centered at the gaze location is for imagery at a high imagequality, and tiles that are adjacent or near or otherwise on the displayare for imagery at a same or lower quality.

The use of the tile centered at the gaze location and the additionaltiles facilitates a foveated display, according to some embodiments. Insome embodiments, imagery of the tile centered at the gaze location andthe additional tiles are rasterized to achieve a foveated display of theimagery. In some embodiments, sizes and/or shapes of the various tilesare adjusted in real-time to account for error associated with the gazedirection of the user's eye. The system can rasterize image data fortiles with lower resolution, lower detail, or lower image quality andupscale (e.g., using nearest neighbor) the rasterized imagery to providea smooth transition between tiles, according to some embodiments. Insome embodiments, tiles that are further away from the gaze location areassociated with lower image quality. Advantageously, the systems andmethods described herein facilitate reduced power consumption ofprocessing circuitry, but still provide detailed imagery within thefovea region, according to some embodiments.

Virtual Reality or Augmented Reality System

Referring now to FIG. 1, a system 100 can include a plurality of sensors104 a . . . n, processing circuitry 116, and one or more displays 164.System 100 can be implemented using HMD system 200 described in greaterdetail below with reference to FIG. 2. System 100 may be configured asan HMD system or a head wearable display (HWD) system. System 100 can beimplemented using the computing environment described with reference toFIG. 4. System 100 can incorporate features of and be used to implementfeatures of virtual reality (VR) systems. At least some of processingcircuitry 116 can be implemented using a graphics processing unit (GPU).The functions of processing circuitry 116 can be executed in adistributed manner using a plurality of processing units.

Processing circuitry 116 may include one or more circuits, processors,and/or hardware components. Processing circuitry 116 may implement anylogic, functions or instructions to perform any of the operationsdescribed herein. Processing circuitry 116 can include any type and formof executable instructions executable by any of the circuits, processorsor hardware components. The executable instructions may be of any typeincluding applications, programs, services, tasks, scripts, librariesprocesses and/or firmware. Any of eye tracker 118, error manager 120,tile generator 122, an image renderer 124 may be any combination orarrangement of circuitry and executable instructions to perform theirrespective functions and operations. At least some portions ofprocessing circuitry 116 can be used to implement image processingexecuted by sensors 104.

Sensors 104 a . . . n can be image capture devices or cameras, includingvideo cameras. Sensors 104 a . . . n may be cameras that generate imagesof relatively low quality (e.g., relatively low sharpness, resolution,or dynamic range), which can help reduce the size, weight, and powerrequirements of system 100. For example, sensors 104 a . . . n cangenerate images having resolutions on the order of hundreds of pixels byhundreds of pixels. At the same time, the processes executed by system100 as described herein can be used to generate display images forpresentation to a user that have desired quality characteristics,including depth characteristics.

Sensors 104 a . . . n (generally referred herein as sensors 104) caninclude any type of one or more cameras. The cameras can be visiblelight cameras (e.g., color or black and white), infrared cameras, orcombinations thereof. Sensors 104 a . . . n can each include one or morelenses 108 a . . . j generally referred herein as lens 108). In someembodiments, sensor 104 can include a camera for each lens 108. In someembodiments, sensor 104 include a single camera with multiple lenses 108a . . . j. In some embodiments, sensor 104 can include multiple cameras,each with multiple lenses 108. The one or more cameras of sensor 104 canbe selected or designed to be a predetermined resolution and/or have apredetermined field of view. In some embodiments, the one or morecameras are selected and/or designed to have a resolution and field ofview for detecting and tracking objects, such as in the field of view ofa HMD for augmented reality. The one or more cameras may be used formultiple purposes, such as tracking objects in a scene or an environmentcaptured by the image capture devices and performing calibrationtechniques described herein.

The one or more cameras of sensor 104 and lens 108 may be mounted,integrated, incorporated or arranged on an HMD to correspond to aleft-eye view of a user or wearer of the HMD and a right-eye view of theuser or wearer. For example, an HMD may include a first camera with afirst lens mounted forward-facing on the left side of the HMDcorresponding to or near the left eye of the wearer and a second camerawith a second lens mounted forward-facing on the right-side of the HMDcorresponding to or near the right eye of the wearer. The left cameraand right camera may form a front-facing pair of cameras providing forstereographic image capturing. In some embodiments, the HMD may have oneor more additional cameras, such as a third camera between the first andsecond cameras an offers towards the top of the HMD and forming atriangular shape between the first, second and third cameras. This thirdcamera may be used for triangulation techniques in performing the depthbuffer generations techniques of the present solution, as well as forobject tracking.

System 100 can include a first sensor (e.g., image capture device) 104 athat includes a first lens 108 a, first sensor 104 a arranged to capturea first image 112 a of a first view, and a second sensor 104 b thatincludes a second lens 108 b, second sensor 104 b arranged to capture asecond image 112 b of a second view. The first view and the second viewmay correspond to different perspectives, enabling depth information tobe extracted from first image 112 a and second image 112 b. For example,the first view may correspond to a left eye view, and the second viewmay correspond to a right eye view. System 100 can include a thirdsensor 104 c that includes a third lens 108 c, third sensor 104 carranged to capture a third image 112 c of a third view. As describedwith reference to FIG. 2, the third view may correspond to a top viewthat is spaced from an axis between first lens 108 a and second lens 108b, which can enable system 100 to more effectively handle depthinformation that may be difficult to address with first sensor 104 a andsecond sensor 104 b, such as edges (e.g., an edge of a table) that aresubstantially parallel to the axis between first lens 108 a and secondlens 108 b.

Light of an image to be captured by sensors 104 a . . . n can bereceived through the one or more lenses 108 a . . . j. Sensors 104 a . .. n can include sensor circuitry, including but not limited tocharge-coupled device (CCD) or complementary metal-oxide-semiconductor(CMOS) circuitry, which can detect the light received via the one ormore lenses 108 a . . . j and generate images 112 a . . . k based on thereceived light. For example, sensors 104 a . . . n can use the sensorcircuitry to generate first image 112 a corresponding to the first viewand second image 112 b corresponding to the second view. The one or moresensors 104 a . . . n can provide images 112 a . . . k to processingcircuitry 116. The one or more sensors 104 a . . . n can provide images112 a . . . k with a corresponding timestamp, which can facilitatesynchronization of images 112 a . . . k when image processing isexecuted on images 112 a . . . k, such as to identify particular firstand second images 112 a, 112 b representing first and second views andhaving the same timestamp that should be compared to one another tocalculate gaze information.

Sensors 104 can include eye tracking sensors 104 or head trackingsensors 104 that can provide information such as positions,orientations, or gaze directions of the eyes or head of the user (e.g.,wearer) of an HMD. In some embodiments, sensors 104 are inside outtracking cameras configured to provide images for head trackingoperations. Sensors 104 can be eye tracking sensors 104 that provide eyetracking data 148, such as data corresponding to at least one of aposition or an orientation of one or both eyes of the user. Sensors 104can be oriented in a direction towards the eyes of the user (e.g., ascompared to sensors 104 that capture images of an environment outside ofthe HMD). For example, sensors 104 can include at least one fourthsensor 104 d (e.g., as illustrated in FIG. 2) which can be orientedtowards the eyes of the user to detect sensor data regarding the eyes ofthe user.

In some embodiments, sensors 104 output images of the eyes of the user,which can be processed to detect an eye position or gaze direction(e.g., first gaze direction) of the eyes. In some embodiments, sensors104 process image data regarding the eyes of the user, and output theeye position or gaze direction based on the image data. In someembodiments, sensors 104 optically measure eye motion, such as byemitting light (e.g., infrared light) towards the eyes and detectingreflections of the emitted light.

As discussed further herein, an eye tracking operation can include anyfunction, operation, routine, logic, or instructions executed by system100 or components thereof to track data regarding eyes of the user, suchas positions or orientations (e.g., gaze directions) of the eyes of theuser as the eyes of the user move during use of the HMD. For example,the eye tracking operation can be performed using at least one of one ormore sensors 104 or eye tracker 118. For example, the eye trackingoperation can process eye tracking data 148 from sensor 104 to determinean eye position, gaze direction, gaze vector, focal point, point ofview, etc., shown as gaze vector 136 of eye(s) of the user. In someembodiments, the eye tracking operation can be performed using eyetracker 118 that is implemented using a portion of processing circuitry116 that is coupled with, mounted to, integral with, implemented using asame circuit board as, or otherwise provided with one or more sensors104 that detect sensor data regarding the eyes of the user. In someembodiments, the eye tracking operation can be performed using an eyetracker 118 that receives sensor data by a wired or wireless connectionfrom the one or more sensors 104 that are configured to detect sensordata regarding the eyes of the user (e.g., images of the eyes of theuser); for example, eye tracker 118 can be implemented using the sameprocessing hardware as at least one of error manager 120, tile generator122, and/or image renderer 124. Various such combinations of sensorhardware of sensors 104 and/or processing hardware of processingcircuitry 116 may be used to implement the eye tracking operation.

Eye tracker 118 can generate gaze vector 136 in various manners. Forexample, eye tracker 118 can process eye tracking data 148 to identifyone or more pixels representing at least one of a position or anorientation of one or more eyes of the user. Eye tracker 118 canidentify, using eye tracking data 148, gaze vector 136 based on pixelscorresponding to light (e.g., light from light sources/light emittingdiodes/actuators of sensors 104, such as infrared or near-infrared lightfrom actuators of sensors 104, such as 850 nm light eye tracking)reflected by the one or more eyes of the user. Eye tracker 118 can uselight from various illumination sources or reflections in the HMD or ARsystem, such as from waveguides, combiners, or lens cameras. Eye tracker118 can determine gaze vector 136 or eye position by determining avector between a pupil center of one or more eyes of the user and acorresponding reflection (e.g., corneal reflection). Gaze vector 136 caninclude position data such as at least one of a position or anorientation of each of one or more eyes of the user. The position datacan be in three-dimensional space, such as three-dimensional coordinatesin a Cartesian, spherical, or other coordinate system. Gaze vector 136can include position data including a gaze direction of one or more eyesof the user. In some embodiments, eye tracker 118 includes a machinelearning model. The machine learning model can be used to generate eyeposition or gaze vector 136 based on eye tracking data 148.

Processing circuitry 116 can include an error manager 120. Error manager120 is configured to receive eye tracking data 148 from sensor(s) 104and determine gaze error 126 associated with gaze vector 136. Gaze error126 can include error for eye position, gaze direction, eye direction,etc., of gaze vector 136 (e.g., gaze location, gaze vector 302, etc.).Error manager 120 can receive eye tracking data 148 from sensor(s) 104and perform an error analysis to determine gaze error 126. Error manager120 monitors eye tracking data 148 over time and/or gaze vector 136 overtime and determines gaze error 126 based on eye tracking data 148 and/orgaze vector 136, according to some embodiments. In some embodiments,error manager 120 provides gaze error 126 to tile generator 122. Eyetracker 118 also provides gaze vector 136 to tile generator 122,according to some embodiments. Error manager 120 can be configured toidentify, determine, calculate, etc., any of rotational velocity,prediction error, fixation error, a confidence interval of gaze vector136, random error, measurement error of gaze vector 136, etc.

Processing circuitry 116 includes tile generator 122, according to someembodiments. Tile generator 122 is configured to receive gaze vector 136from eye tracker 118 and gaze error 126 from error manager 120,according to some embodiments. Tile generator 122 is configured todefine one or more tiles 128 (e.g., tiles 602 shown in FIGS. 6-15 and21), superpixels, collection of pixels, render areas, resolution areas,etc., for image renderer 124, according to some embodiments. Tilegenerator 122 generates tiles 128 based on gaze vector 136, a focal gazelocation of the user's eyes, a reference gaze location, a direction ofgaze, eye position, a point of interest, etc., according to someembodiments. Tile generator 122 generates various subsets of tiles 128for display on display(s) 164 and corresponding resolutions, accordingto some embodiments. In some embodiments, tile generator 122 defines afirst set of tiles 128 that should have a high resolution (e.g., a highlevel of detail, high image quality, etc.), a second set of tiles 128that should have a medium resolution, and a third set of tiles thatshould have a low resolution. Tiles 128 include a corresponding size(e.g., height and width, number of pixels, gaze angles, etc.) for eachtile 128, according to some embodiments.

In some embodiments, tiles 128 include data regarding a correspondingposition on display(s) 164. For example, tile generator 122 generatesmultiple tiles 128 that collectively cover an entirety of display(s) 164and associated positions within display(s) 164, according to someembodiments. Tile generator 122 provides tiles 128 to image renderer 124for use in generating a rendered image 130, according to someembodiments. Tile generator 122 also generates or defines tiles 128based on gaze error 126, according to some embodiments. In someembodiments, tile generator 122 divides a total area of display(s) 164into various subsections, collection of pixels, etc., referred to astiles 128. Tile generator 122 assigns a corresponding resolution to eachof tiles 128, according to some embodiments. In some embodiments, tilegenerator 122 redefines tiles 128 periodically or dynamically based onupdated or new gaze error 126 and/or gaze vector 136. In someembodiments, tile generator 122 defines a size, shape, position, andcorresponding resolution of imagery for each of tiles 128. In someembodiments, any of the size, position, and corresponding resolution ofimagery for each of tiles 128 is determined by tile generator 122 basedon gaze vector 136 and/or gaze error 126.

Processing circuitry 116 includes image renderer 124, according to someembodiments. In some embodiments, image renderer 124 is configured toreceive tiles 128 from tile generator 122 and use tiles 128 to generatean image for display(s) 164. In some embodiments, image renderer 124receives image data 132 and uses tiles 128 to display the image data ondisplay(s) 164. In some embodiments, image renderer 124 receives tiles128 and image data 132 and generates a rendered image 130 based on tiles128 and image data 132. Image renderer 124 uses the size, shape,position, and corresponding resolution of each of tiles 128 to rasterizeimage data 132 to generate rendered image 130, according to someembodiments.

Image renderer 124 is a 3D image renderer or 2D image renderer,according to some embodiments. Image renderer 124 uses image relatedinput data to process, generate and render display or presentationimages to display or present on one or more display devices, such as viaan HMD, according to some embodiments. Image renderer 124 generates orcreates 2D images of a scene or view for display on display 164 andrepresenting the scene or view in a 3D manner, according to someembodiments. The display or presentation data (e.g., image data 132) tobe rendered includes geometric models of 3D objects in the scene orview, according to some embodiments. Image renderer 124 determines,computes, or calculates the pixel values of the display or image data tobe rendered to provide the desired or predetermined 3D image(s), such as3D display data for images 112 captured by the sensor 104, according tosome embodiments. Image renderer 124 receives images 112, tiles 128, andhead tracking data 150 and generates display images using images 112.

Image renderer 124 can render frames of display data to one or moredisplays 164 based on temporal and/or spatial parameters. Image renderer124 can render frames of image data sequentially in time, such ascorresponding to times at which images are captured by the sensors 104.Image renderer 124 can render frames of display data based on changes inposition and/or orientation to sensors 104, such as the position andorientation of the HMD. Image renderer 124 can render frames of displaydata based on left-eye view(s) and right-eye view(s) such as displayinga left-eye view followed by a right-eye view or vice-versa.

Image renderer 124 can generate the display images using motion dataregarding movement of the sensors 104 a . . . n that captured images 112a . . . k. For example, the sensors 104 a . . . n may change in at leastone of position or orientation due to movement of a head of the userwearing an HMD that includes the sensors 104 a . . . n (e.g., asdescribed with reference to HMD system 200 of FIG. 2). Processingcircuitry 116 can receive the motion data from a position sensor (e.g.,position sensor 220 described with reference to FIG. 2). Image renderer124 can use the motion data to calculate a change in at least one ofposition or orientation between a first point in time at which images112 a . . . k were captured and a second point in time at which thedisplay images will be displayed, and generate the display images usingthe calculated change. Image renderer 124 can use the motion data tointerpolate and/or extrapolate the display images relative to images 112a . . . k. Although image renderer 124 is shown as part of processingcircuitry 116, the image renderer may be formed as part of otherprocessing circuitry of a separate device or component, such as thedisplay device, for example within the HMD.

System 100 can include one or more displays 164. The one or moredisplays 164 can be any type and form of electronic visual display. Thedisplays may have or be selected with a predetermined resolution andrefresh rate and size. The one or more displays can be of any type oftechnology such as LCD, LED, ELED or OLED based displays. The formfactor of the one or more displays may be such to fit within the HMD asglasses or goggles in which the display(s) are the lens within the frameof the glasses or goggles. Displays 164 may have a refresh rate the sameor different than a rate of refresh or frame rate of processingcircuitry 116 or image renderer 124 or the sensors 104.

Referring now to FIG. 2, in some implementations, an HMD system 200 canbe used to implement system 100. HMD system 200 can include an HMD body202, a left sensor 104 a (e.g., left image capture device), a rightsensor 104 b (e.g., right image capture device), and display 164. HMDbody 202 can have various form factors, such as glasses or a headset.The sensors 104 a, 104 b can be mounted to or integrated in HMD body202. The left sensor 104 a can capture first images corresponding to afirst view (e.g., left eye view), and the right sensor 104 b can captureimages corresponding to a second view (e.g., right eye view). HMD system200 may also be a HWD system.

HMD system 200 can include a top sensor 104 c (e.g., top image capturedevice). Top sensor 104 c can capture images corresponding to a thirdview different than the first view or the second view. For example, topsensor 104 c can be positioned between the left sensor 104 a and rightsensor 104 b and above a baseline between the left sensor 104 a andright sensor 104 b. This can enable top sensor 104 c to capture imageswith depth information that may not be readily available to be extractedfrom the images captured by left and right sensors 104 a, 104 b. Forexample, it may be difficult for depth information to be effectivelyextracted from images captured by left and right sensors 104 a, 104 b inwhich edges (e.g., an edge of a table) are parallel to a baselinebetween left and right sensors 104 a, 104 b. Top sensor 104 c, beingspaced from the baseline, can capture the third image to have adifferent perspective, and thus enable different depth information to beextracted from the third image, than left and right sensors 104 a, 104b.

HMD system 200 can include processing circuitry 116, which can performat least some of the functions described with reference to FIG. 1,including receiving sensor data from sensors 104 a, 104 b, and 104 c aswell as eye tracking sensors 104, and processing the received images tocalibrate an eye tracking operation.

HMD system 200 can include communications circuitry 204. Communicationscircuitry 204 can be used to transmit electronic communication signalsto and receive electronic communication signals from at least one of aclient device 208 or a server 212. Communications circuitry 204 caninclude wired or wireless interfaces (e.g., jacks, antennas,transmitters, receivers, transceivers, wire terminals) for conductingdata communications with various systems, devices, or networks. Forexample, communications circuitry 204 can include an Ethernet card andport for sending and receiving data via an Ethernet-based communicationsnetwork. Communications circuitry 204 can communicate via local areanetworks (e.g., a building LAN), wide area networks (e.g., the Internet,a cellular network), and/or conduct direct communications (e.g., NFC,Bluetooth). Communications circuitry 204 can conduct wired and/orwireless communications. For example, communications circuitry 204 caninclude one or more wireless transceivers (e.g., a Wi-Fi transceiver, aBluetooth transceiver, a NFC transceiver, a cellular transceiver). Forexample, communications circuitry 204 can establish wired or wirelessconnections with the at least one of the client device 208 or server212. Communications circuitry 204 can establish a USB connection withthe client device 208.

HMD system 200 can be deployed using different architectures. In someembodiments, the HMD (e.g., HMD body 202 and components attached to HMDbody 202) comprises processing circuitry 116 and is self-containedportable unit. In some embodiments, the HMD has portions of processingcircuitry 116 that work in cooperation with or in conjunction with anytype of portable or mobile computing device or companion device that hasthe processing circuitry or portions thereof, such as in the form of astaging device, a mobile phone or wearable computing device. In someembodiments, the HMD has portions of processing circuitry 116 that workin cooperation with or in conjunction with processing circuitry, orportions thereof, of a desktop computing device. In some embodiments,the HMD has portions of processing circuitry 116 that works incooperation with or in conjunction with processing circuitry, orportions thereof, of a server computing device, which may be deployedremotely in a data center or cloud computing environment. In any of theabove embodiments, the HMD or any computing device working inconjunction with the HMD may communicate with one or more servers inperforming any of the functionality and operations described herein.

The client device 208 can be any type and form of general purpose orspecial purpose computing device in any form factor, such as a mobile orportable device (phone, tablet, laptop, etc.), or a desktop or personalcomputing (PC) device. In some embodiments, the client device can be aspecial purpose device, such as in the form of a staging device, whichmay have the processing circuitry or portions thereof. The specialpurpose device may be designed to be carried by the user while wearingthe HMD, such as by attaching the client device 208 to clothing or thebody via any type and form of accessory attachment. The client device208 may be used to perform any portion of the image and renderingprocessing pipeline described in connection with FIGS. 1 and 3. The HMDmay perform some or other portions of the image and rendering processingpipeline such as image capture and rendering to display 164. The HMD cantransmit and receive data with the client device 208 to leverage theclient device 208's computing power and resources which may have higherspecifications than those of the HMD.

Server 212 can be any type of form of computing device that providesapplications, functionality or services to one or more client devices208 or other devices acting as clients. In some embodiments, server 212can be a client device 208. Server 212 can be deployed in a data centeror cloud computing environment accessible via one or more networks. TheHMD and/or client device 208 can use and leverage the computing powerand resources of server 212. The HMD and/or client device 208 canimplement any portion of the image and rendering processing pipelinedescribed in connection with FIGS. 1 and 3. Server 212 can implement anyportion of the image and rendering processing pipeline described inconnection with FIGS. 1 and 3, and in some cases, any portions of theimage and rendering processing pipeline not performed by client device208 or HMD. Server 212 may be used to update the HMD and/or clientdevice 208 with any updated to the applications, software, executableinstructions and/or data on the HMD and/or client device 208.

System 200 can include a position sensor 220. The position sensor 220can output at least one of a position or an orientation of the body 202.As the image capture devices 104 a, 104 b, 104 c can be fixed to thebody 202 (e.g., at predetermined locations relative to the positionsensor 220), the position sensor 220 can output at least one of aposition or an orientation of each sensor 104 a, 104 b, 104 c. Theposition sensor 220 can include at least one of an inertial measurementunit (IMU), an accelerometer, a gyroscope, or a magnetometer (e.g.,magnetic compass).

System 200 can include a varifocal system 224. Varifocal system 224 canhave a variable focal length, such that varifocal system 224 can changea focus (e.g., a point or plane of focus) as focal length ormagnification changes. Varifocal system 224 can include at least one ofa mechanical lens, liquid lens, or polarization beam plate. In someembodiments, varifocal system 224 can be calibrated by processingcircuitry 116 (e.g., by a calibrator), such as by receiving anindication of a vergence plane from a calibrator which can be used tochange the focus of varifocal system 224. In some embodiments, varifocalsystem 224 can enable a depth blur of one or more objects in the sceneby adjusting the focus based on information received from the calibratorso that the focus is at a different depth than the one or more objects.

In some embodiments, display 164 includes one or more waveguides. Thewaveguides can receive (e.g., in-couple) light corresponding to displayimages to be displayed by display 164 from one or more projectors, andoutput (e.g., out-couple) the display images, such as for viewing by auser of the HMD. The waveguides can perform horizontal or verticalexpansion of the received light to output the display images at anappropriate scale. The waveguides can include one or more lenses,diffraction gratings, polarized surfaces, reflective surfaces, orcombinations thereof to provide the display images based on the receivedlight. The projectors can include any of a variety of projectiondevices, such as LCD, LED, OLED, DMD, or LCOS devices, among others, togenerate the light to be provided to the one or more waveguides. Theprojectors can receive the display images from processing circuitry 116(e.g., from image renderer 124). The one or more waveguides can beprovided through a display surface (e.g., glass), which can be at leastpartially transparent to operate as a combiner (e.g., combining lightfrom a real world environment around the HMD with the light of theoutputted display images).

Display 164 can perform foveated rendering based on the calibrated eyetracking operation, which can indicate a gaze point corresponding to thegaze direction generated by the eye tracking operation. For example,processing circuitry 116 can identify at least one of a central regionof the FOV of display 164 (e.g., a plurality of pixels within athreshold distance from the gaze point) peripheral region of the FOV ofdisplay 164 based on the gaze point (e.g., a peripheral regionrepresented by a plurality of pixels of the display images that arewithin a threshold distance of an edge of the display images or morethan a threshold distance from the gaze point). Processing circuitry 116can generate the display images to have a less quality (e.g.,resolution, pixel density, frame rate) in the peripheral region than inthe central region, which can reduce processing demand associated withoperation of HMD system 200.

Gaze Vector and Point of Interest

Referring now to FIGS. 3-5, the gaze vector is shown in greater detail,according to some embodiments. Gaze vector 136 as used by processingcircuitry 116 is represented graphically in FIGS. 3-5 as gaze vector302, according to some embodiments. It should be understood that whilegaze vector 136 is represented in a spherical coordinate system, gazevector 136 can also be represented in a Cartesian coordinate system, apolar coordinate system, a cylindrical coordinate system, etc., or anyother coordinate system. Gaze vector 302 is used by processing circuitry116 to determine a focal point or gaze location 402 of the user's eyes,according to some embodiments.

Referring particularly to FIG. 3, a spherical coordinate system includesgaze vector 302, and a user's eye (or eyes) 140. Eye 140 is shown as acenterpoint of the spherical coordinate system, and gaze vector 302extends radially outwards from eye 140, according to some embodiments.In some embodiments, a direction of gaze vector 302 is defined by one ormore angles, shown as angle θ₁ and angle θ₂. In some embodiments, angleθ₁ represents an angular amount between gaze vector 302 and a verticalaxis 304. In some embodiments, angle θ₂ represents an angular amountbetween gaze vector 302 and a horizontal axis 306. In some embodiments,vertical axis 304 and horizontal axis 306 are substantiallyperpendicular to each other and both extend through eye 140.

In some embodiments, eye tracker 118 of processing circuitry 116 isconfigured to determine values of both angle θ₁ and angle θ₂ based oneye tracking data 148. Eye tracker 118 can determine the values ofangles θ₁ and θ₂ for both eyes 140, according to some embodiments. Insome embodiments, eye tracker 118 determines the values of angles θ₁ andθ₂ and provides the angles to error manager 120 and/or tile generator122 as gaze vector 136.

Referring particularly to FIGS. 4 and 5 gaze vector 302 can be used todetermine a location of a point of interest, a focal point, a gazepoint, a gaze location, a point, etc., shown as gaze location 402. Gazelocation 402 has a location on display 164, according to someembodiments. In some embodiments, gaze location 402 has an x locationand a y location (e.g., a horizontal and a vertical location) on display164. In some embodiments, gaze location 402 has a location in virtualspace, real space, etc. In some embodiments, gaze location 402 has a twodimensional location. In some embodiments, gaze location 402 has athree-dimensional location. Gaze location 402 can have a location ondisplay 164 relative to an origin or a reference point on display 164(e.g., a center of display 164, a corner of display 164, etc.). Gazelocation 402 and gaze vector 302 can be represented using any coordinatesystem, or combination of coordinate systems thereof. For example, gazelocation 402 and/or gaze vector 302 can be defined using a Cartesiancoordinate system, a polar coordinate system, a cylindrical coordinatesystem, a spherical coordinate system, a homogeneous coordinate system,a curvilinear coordinate system, an orthogonal coordinate system, a skewcoordinate system, etc.

In some embodiments, tile generator 122 and/or eye tracker 118 areconfigured to use a distance d between the user's eye 140 and display164. The distance d can be a known or sensed distance between the user'seye 140 and display 164, according to some embodiments. For example,sensors 104 can measure, detect, sense, identify, etc., the distance dbetween the user's eye 140 and display 164. In some embodiments, thedistance d is a known distance based on a type or configuration of theHMD.

The distance d and the angles θ₁ and θ₂ can be used by eye tracker 118to determine gaze vector 302/136. In some embodiments, eye tracker 118uses the distance d and the angles θ₁ and θ₂ to determine the locationof gaze location 402. In some embodiments, eye tracker 118 provides thedistance d and the angles θ₁ and κ₂ to tile generator 122. Tilegenerator 122 uses the distance d and the angles θ₁ and θ₂ to determinethe location of gaze location 402 relative to a reference point ondisplay 164.

FIG. 4 is a top view of display 164 and the user's eye 140, according tosome embodiments. FIG. 4 shows the angle θ₁, according to someembodiments. Likewise, FIG. 5 is a side view of display 164 and theuser's eye 140 and shows the angle θ₂, according to some embodiments.Tile generator 122 and/or eye tracker 118 use the distance d and theangles θ₁ and θ₂ to determine the position/location of gaze location402, according to some embodiments. In some embodiments, tile generator122 uses the position/location of gaze location 402 to define tiles 128.It should be understood that while display 164 is shown as a generallyflat display screen, in some embodiments, display 164 is a curved,arcuate, etc., display screen. A rectangular display screen is shown forease of illustration and description only. Accordingly, all referencesto “local positions,” “local coordinates,” “Cartesian coordinates,”etc., of display 164 may refer to associated/corresponding angularvalues of angle θ₁ and/or angle θ₂.

Tile Definition

Referring to FIG. 6, display 164 is shown to include tiles 602,according to some embodiments. In some embodiments, tiles 602 aredefined by tile generator 122 based on the location/position of gazelocation 402. Gaze location 402 represents an approximate location thatthe user is viewing, according to some embodiments. In some embodiments,gaze location 402 represents the point or location that the user's gazeis directed towards.

Display 164 includes tiles 602 having a width w and a height h,according to some embodiments. In some embodiments, the width w isreferred to as a length along a central horizontal axis of display 164(e.g., a straight horizontal axis if display 164 is straight, a curvedhorizontal axis if display 164 is curved). Likewise, the height h isreferred to as a height along a vertical axis of display 164 (e.g., astraight vertical axis if display 164 is straight, a curved verticalaxis if display 164 is curved about horizontal axis 306). In someembodiments, the width w and the height h are angular values of angle θ₁and θ₂ for a given distance from the user's eye 140. For example, thewidth w of tiles 602 may be an 11 degrees (e.g., an amount of 11 degreesfor angle θ₁ from opposite sides of tile 602), and the height h of tiles602 may be 17 degrees (e.g., an amount of 17 degrees for angle θ₂ fromtop and bottom sides of tile 602).

In some embodiments, each of tiles 602 have an area A=wh. In someembodiments, each of tiles 602 includes a collection of pixels thatdisplay a portion of an image that is displayed on display 164 to theuser. Tiles 602 collectively display the image to the user on display164, according to some embodiments. The image can be a rendered image ofthree dimensional objects, particles, characters, terrain, maps, text,menus, etc. In some embodiments, the image is a virtual reality image.In some embodiments, the image is an augmented reality image (e.g.,imagery is overlaid or projected over a real-world image). For example,if display 164 is a display of a HMD virtual reality system, the imagecan be a representation of a virtual reality, a virtual space, a virtualenvironment, etc. Likewise, if display 164 is a display of a HMDaugmented reality system, the image can be a representation of projectedobjects, characters, particles, text, etc., having a location in virtualspace that matches or corresponds or tracks a location in real space.

Referring still to FIG. 6, display 164 includes a first set of tiles 602a, a second set of tiles 602 b, and a third set of tiles 602 c,according to some embodiments. In some embodiments, the resolution oftiles 602 c is greater than the resolution of tiles 602 b, and theresolution of tiles 602 b is greater than the resolution of tiles 602 a.In some embodiments, processing power of processing circuitry 116 can bereduced by decreasing the resolution of tiles 602 that are in the user'speripheral view. For example, tiles 602 that are currently being viewedout of the corner of the user's eye may be rendered at a lowerresolution without the user noticing the reduced or lower resolution.

Tile generator 122 determines which tiles 602 should be rendered byimage renderer 124 at a higher resolution, according to someembodiments. In some embodiments, tile generator 122 includes apredetermined number of tile groups with ascending or descendingresolutions. Tile generator 122 can monitor the location/position ofgaze location 402 in real-time (e.g., before every frame is rendered andprovided to the user on display 164) to determine which of tiles 602should be rendered at the highest resolution and to determine which oftiles 602 should be rendered in a lower resolution to save processingpower of processing circuitry 116. In some embodiments, tile generator122 determines that tiles 602 which correspond to the position/locationof gaze location 402 should have the highest resolution. For example, ifgaze location 402 is located within a specific tile 602, tile generator122 can determine that the specific tile 602 should have the highestresolution or should be associated with the highest resolution set oftiles 602. In some embodiments, tile generator 122 also determines thattiles 602 that are adjacent, neighboring, or nearby (e.g., directlyadjacent) the specific tile 602 should also be rendered with the samehigh resolution (e.g., tiles 602 a). In some embodiments, tile generator122 determines that tiles 602 that are adjacent, next to, neighbor,nearby, etc., the high resolution tiles 602 should be rendered at amedium resolution (e.g., tiles 602 b). Tile generator 122 can determinethat all other tiles 602 of display 164 which are in the user'speripheral view should be rendered at the low resolution (e.g., tiles602 c).

In some embodiments, tile generator 122 adjusts which of tiles 602should be rendered at the high resolution, the medium resolution, andthe lower resolution. For example, tile generator 122 can re-assigntiles 602 to redefine the various tile groups in response to thelocation/position of gaze location 402 changing. For example, as shownin FIGS. 6 and 7, the location/position of gaze location 402 on display164 changes as the user re-directs their gaze, according to someembodiments. Tile generator 122 redefines the first set of tiles 602 aso that tiles 602 that are near or at the location/position of gazelocation 402 are rendered at the highest quality/resolution, accordingto some embodiments. Likewise, tile generator 122 redefines the secondset of tiles 602 b and the third set of tiles 602 c to account for theshift in the location/position of gaze location 402. In this way, tilegenerator 122 can redefine the resolution of each of tiles 602 inresponse to the user changing or redirecting their gaze. In someembodiments, tile generator 122 uses gaze vector 302 in real-time tore-calculate the location/position of gaze location 402. Tile generator122 uses the re-calculated or updated location/position of gaze location402 to redefine or reassign tiles 602 (e.g., to re-determine positions,sizes, shapes, definitions, locations, resolutions, etc., of tiles 602).

Referring now to FIGS. 8 and 9, tile generator 122 uses gaze error 126to adjust or redefine tiles 602, according to some embodiments. Gazeerror 126 is represented by error 604, according to some embodiments.Error 604 indicates an area or range of locations that gaze location 402may be located, according to some embodiments. In some embodiments,error 604 is determined based on eye tracking data 148 by error manager120. In some embodiments, error 604 is used by tile generator 122 todetermine if the user's gaze may be directed to an area or locationother than the location/position of gaze location 402. Error 604represents an area that gaze location 402 may be located, correspondingto an error of angle θ₁ and angle θ₂, according to some embodiments. Forexample, if angle θ₁ has a high amount of error, error 604 (representedby a two-dimensional area on display 164) may have a larger height,according to some embodiments. Likewise, if angle θ₂ has a high amountof error, error 604 may have a larger width, according to someembodiments. Error 604 indicates angular jitter, wobble, measurementuncertainty, etc., of gaze vector 302 (and/or gaze location 402) aboutvertical axis 304 and/or horizontal axis 306, according to someembodiments.

If error 604 increases, a corresponding number of tiles 602 that theuser's gaze may be directed towards also increases, according to someembodiments. In response to error 604 increasing, tile generator 122 canredefine the first set of tiles 602 a, the second set of tiles 602 b,and the third set of tiles 602 c to account for the increased error 604.For example, as shown in FIG. 8, five tiles 602 are included in thefirst set of tiles 602 a and are rendered at the high resolution,according to some embodiments. However, as error 604 increases (shown inFIGS. 8 and 9), tile generator 122 includes additional tiles 602 toensure that the user's gaze is not directed towards a lower resolutiontile 602, according to some embodiments. Error manager 120 can receiveeye tracking data 148 in real-time, determine gaze error 126, translategaze error 126 (e.g., the error of angle θ₁ and/or angle θ₂) to a rangeof positions/locations on display 164 (shown as error 604), and provideerror 604 and/or gaze error 126 to tile generator 122. Tile generator122 defines the size, area, number of pixels, location, resolution,etc., of each tile 602 based on gaze error 126 (e.g., based on error604), according to some embodiments. In some embodiments, tile generator122 provides any of the defined information regarding tiles 602 to imagerenderer 124. In some embodiments, tile generator 122 uses both thelocation/position of gaze location 402 as well as error 604 to definetiles 602. In some embodiments, the positions, locations, sizes, etc.,of tiles 602 are predefined or predetermined, and tile generator 122defines a corresponding resolution for each tile 602 based on thelocation/position of gaze location 402 as well as error 604.

For example, if angle θ₁ has an error Δθ₁ and angle θ₂ has an error Δθ₂,error manager 120 determines an error Δx in the x position of gazelocation 402, and an error Δy in the y position of gaze location 402,according to some embodiments. In some embodiments, the error Δx and theerror Δy define error 604. In some embodiments, error manager 120translates the uncertainty, error, range of values, etc., of the anglesθ₁ and θ₂ to errors Δx and Δy in the x direction and they direction ofdisplay 164 to define error 604.

In some embodiments, error 604 has the shape of a circle (as shown inFIGS. 6-15). In some embodiments, error 604 has the shape of an ellipse,a square, a rectangle, etc. Tile generator 122 can identify, based onthe location/position of gaze location 402 and error 604, which tiles602 lie within error 604 (e.g., if tiles 602 havepredetermined/predefined locations and sizes) and assigns these tiles602 with high resolution rendering. If error 604 increases, additionaltiles 602 that now are within error 604 can be re-assigned by tilegenerator 122 to render with high resolution. Likewise, if error 604decreases, tiles 602 that are not within error 604 can be re-assigned bytile generator 122 to render with lower resolution. In this way,foveated rendering can be achieved that accounts for error in the gazevector, gaze direction, point of interest, focal point, etc., of theuser.

In some embodiments, display 164 includes smaller tiles 602 as shown inFIGS. 10-11. The height h and width w of tiles 602 are defined by tilegenerator 122 based on display capabilities of display 164, according tosome embodiments. In some embodiments, the height h and width w of tiles602 are predefined for display 164. For example, displays that arecapable of displaying higher resolution images (e.g., displays with morepixels) can have additional tiles 602 compared to displays with lowerresolution (e.g., less pixels). Tile generator 122 can dynamicallyre-define positions, sizes, shapes, etc., of each of tiles 602 andcorresponding render/resolution qualities in response to gaze location402 shifting (e.g., shifting to the right as shown in FIGS. 10-11).

Referring particularly to FIGS. 12-13, gaze location 402 may sometimesremain within a corresponding tile 602, according to some embodiments.However, gaze location 402 can also move near an intersection ofmultiple tiles 602. For example, if gaze location 402 moves to theintersection between the four tiles 602 shown in FIG. 12, a situationarises in which tile generator 122 must determine which of the fourtiles 602 should be rendered in the high resolution, according to someembodiments. Tile generator 122 can render all four tiles 602 in thehigh resolution, however, this may not necessarily be the mostprocessing efficient solution.

In cases when gaze location 402 moves near an intersection of one ormore tiles 602, tile generator 122 can redefine a layout, size,position, arrangement, etc., of tiles 602. In some embodiments, tilegenerator 122 redefines the layout, size, position, arrangement, etc.,of tiles 602 in response to gaze location 402 moving a predetermineddistance from a centerpoint of a tile 602. For example, if gaze location402 is within a predetermined distance from the centerpoint of thebottom left tile 602 as shown in FIG. 12, tile generator 122 maintainsthe current layout, size, position, arrangement, etc., of tiles 602,according to some embodiments. However, if gaze location 402 moves thepredetermined distance or more from the centerpoint of the bottom lefttile 602 shown in FIG. 12, tile generator 122 redefines the layout,size, position, location, arrangement, etc., of tiles 602, according tosome embodiments. In some embodiments, the predetermined distance iszero (e.g., if gaze location 402 moves from the centerpoint of tile 602any amount, tile generator 122 redefines tiles 602). In someembodiments, the predetermined distance is a number of pixels, anangular amount for a given distance from the user's eye 140 (e.g., anamount of angular rotation about either vertical axis 304 and/orhorizontal axis 306), a distance (e.g., in x and y directions) ondisplay 164, etc. In some embodiments, the predetermined distance is aportion of, a percentage of, or is proportional to a height h or a widthw of a tiles 602 that gaze location 402 is currently within. Thepredetermined distance includes both x and y components or correspondingangular amounts, according to some embodiments.

In some embodiments, tile generator 122 redefines tiles 602 in responseto an intersection (e.g., a border, a boundary, etc.) of one or moretiles 602 lying within error 604. For example, gaze location 402 asshown in FIG. 12 includes error 604, but none of the intersections ofadjacent tiles 602 lie within error 604, according to some embodiments.However, when gaze location 402 moves to the position shown in FIG. 13,the intersections or borders of adjacent tiles 602 are now within error604, according to some embodiments. In response to gaze location 402moving off center of a corresponding tile 602 that gaze location 402 iscurrently within, or in response to an intersection, border, boundary,etc., of the corresponding tile 602 lying within error 604, tilegenerator 122 redefines the layout, size, position, arrangement, etc.,of tiles 602, according to some embodiments. Tile generator 122redefines the layout, size, positions, arrangements, etc., of tiles 602such that gaze location 402 is at a centerpoint of a corresponding tile602, according to some embodiments. Advantageously, this allowsprocessing circuitry 116 to provide high detail or high resolutionimagery on new tile 602 n instead of displaying high detail/highresolution imagery on all four of tiles 602, thereby decreasingprocessing power requirements.

In the example shown in FIGS. 12-13, tile generator 122 defines a newtile 602 n that is centered about the location/position of gaze location402, according to some embodiments. Tile generator 122 assigns new tile602 n the highest resolution for rendering, according to someembodiments.

In some embodiments, tile generator 122 generates, defines, determinesdefinitions of, etc., tiles 602 for the rest of display 164 based on newtile 602 n. Tile generator 122 uses any of the techniques,functionality, etc., described in greater detail above with reference toFIGS. 6-11 to assign or map resolution qualities to the rest of tiles602 that surround new tile 602 n and fill out the entirety of display164, according to some embodiments. New tile 602 n is defined as beingpositioned centrally at gaze location/position 402, according to someembodiments. In some embodiments, a size of new tile 602 n is very smallcompared to gaze angles θ₁ and/or θ₂ or gaze angles θ₁ and/or θ₂ arevery large compared to the size of new tile 602 n. If the size of newtile 602 n is very small compared to gaze angles θ₁ and/or θ₂, new tile602 n can include multiple tiles (e.g., 2 tiles, 4 tiles, etc.). In someembodiments, new tile 602 n does not have a fixed size or ratio betweenheight and width (e.g., new tile 602 n can be a square tile or arectangular tile). In some embodiments, surrounding tiles 602 can bedifferent shapes and/or different sizes (e.g., differently sized squaresand/or rectangles).

In some embodiments, tile generator 122 defines new tile 602 n havingheight h_(new) and width w_(new) that are the same as height h and widthw of previously defined tiles 602. In some embodiments, the heighth_(new) and width w_(new) of new tile 602 n are predetermined values. Insome embodiments, tiles 602 all have uniform height h and width w. Insome embodiments, tiles 602 have non-uniform heights h and widths w. Insome embodiments, the height h and/or width w of tiles 602 are definedby tile generator 122 based on error 604 and/or based on motion of gazelocation 402. For example, if error 604 is large, the height h_(new) andwidth w_(new) of new tile 602 n may also increase, according to someembodiments. In some embodiments, the height h_(new) and the widthw_(new) of new tile 602 n is dynamically related to any of error 604, arate of change of error 604, a speed of motion of gaze location 402,etc.

For example, the height h_(new) of new tile 602 n (or any other tiles602) is related to a vertical error component of error 604:h=f(e _(vertical))where h is the height of new tile 602 n (or any other tile), ande_(vertical) is a vertical error component of error 604 (e.g., inCartesian coordinates, in terms of θ₂, etc.) according to someembodiments. For example, the height h of new tile 602 n or any othertile may be:h=c*e _(vertical)where c is a constant (e.g., an integer such as 1, 2, 3, 4, . . . , orany other non-integer value such as 1.5, 2.5, etc.).

The height h_(new) of new tile 602 n (or any other tiles 602) is relatedto a rate of change of the vertical error component of error 604:h=f(ė _(vertical))according to some embodiments.

The height h_(new) of new tile 602 n is related to the vertical positionof gaze location 402 (described in greater detail below with referenceto FIG. 21):h=f(p _(vertical))where p_(vertical) is a vertical position of gaze location 402 (e.g., ay-position on display 164, a value of angle θ₂, etc.) according to someembodiments.

The height h_(new) of new tile 602 n is related to a rate of change thevertical position of gaze location 402:h=f({dot over (p)} _(vertical))where p_(vertical) is a rate of change (e.g., a time rate of change) ofthe vertical position of gaze location 402 (e.g., a y-position ondisplay 164, a value of angle θ₂, etc.) according to some embodiments.

In some embodiments, the height h of any of tiles 602 (e.g., h_(new),the height of new tile 602 n, or the height of any other tile 602) isrelated to any combination of e_(vertical), ė_(vertical), p_(vertical),and/or {dot over (p)}_(vertical). For example, the height h can berelated to or determined by tile generator 122 using a function,equation, set of equations, etc., such as;h=f(e _(vertical) ,ė _(vertical) ,{dot over (p)} _(vertical) ,p_(vertical))according to some embodiments.

Likewise, the width w_(new) of new tile 602 n (or any other tiles 602)is related to a horizontal error component of error 604:w=f(e _(horizontal))where w is the width of new tile 602 n (or any other tile), ande_(horizontal) is a horizontal component of error 604 (e.g., inCartesian coordinates, in terms of θ₁, etc.) according to someembodiments. For example, the width h of new tile 602 n or any othertile may be:w=c*e _(horizontal)where c is a constant (e.g., an integer such as 1, 2, 3, 4, . . . , orany other non-integer value such as 1.5, 2.5, etc.).

The width w_(new) of new tile 602 n (or any other tiles 602) is relatedto a rate of change (e.g., a time rate of change) of the horizontalcomponent of error 604:w=f(ė _(horizontal))according to some embodiments.

The width w_(new) of new tile 602 n is related to the horizontalposition of gaze location 402 (described in greater detail below withreference to FIG. 21):h=f(p _(horizontal))where p_(horizontal) is a horizontal position of gaze location 402(e.g., an x-position on display 164, a value of angle θ₁, etc.)according to some embodiments.

The width w_(new) of new tile 602 n is related to a rate of change thehorizontal position of gaze location 402:h=f({dot over (p)} _(horizontal))where p_(horizontal) is a rate of change (e.g., a time rate of change)of the horizontal position of gaze location 402 (e.g., an x-position ondisplay 164, a value of angle θ₁, etc.) according to some embodiments.

In some embodiments, the width w of any of tiles 602 (e.g., w_(new), theheight of new tile 602 n, or the height of any other tile 602) isrelated to any combination of e_(horizontal), ė_(horizontal),p_(horizontal), and/or {dot over (p)}_(horizontal). For example, thewidth w can be related to or determined by tile generator 122 using afunction, equation, set of equations, etc., such as;w=f(e _(horizontal) ,ė _(horizontal) ,{dot over (p)} _(horizontal) ,p_(horizontal))according to some embodiments.

In this way, tile generator 122 dynamically redefines the size, shape,dimensions, height, width, arrangement, layout, resolution, etc., oftiles 602 to maintain gaze location 402 within a corresponding tile 602or to maintain gaze location 402 at a centerpoint of the correspondingtile 602, according to some embodiments. Advantageously, tile generator122 redefines all tiles 602 that are used to render images on display164 and dynamically provides foveated rendering, according to someembodiments. Tile generator 122 facilitates reducing processing power ofprocessing circuitry 116 by assigning lower resolution rendering totiles 602 that are in the user's peripheral view, according to someembodiments. However, tiles 602 that are at, adjacent, near,neighboring, etc., gaze location 402 are rendered in higher quality,according to some embodiments.

In some embodiments, tile generator 122 stores multiple layouts of tiles602 that are predetermined or predefined in a database for multiple gazepositions/locations 402. For example, tile generator 122 can store alayout of tiles 602 that are optimized to eye tracking performanceacross all gaze angles. In some embodiments, tile generator 122 uses thepredetermined/predefined layouts dynamically. For example, if gazelocation 402 is at a first point or in a first area, tile generator 122can retrieve and use a corresponding layout of tiles 602. Likewise, ifgaze location 402 changes to a second point or a second area, tilegenerator 122 can retrieve and use a different corresponding layout oftiles 602.

In some embodiments, tile generator 122 generates transitional tiles 602between tiles 602 with different render qualities. The transitionaltiles 602 may be smaller and may provide imagery at intermediatequalities. For example, if a low quality tile 602 is directly adjacent ahigh quality tile 602, transitional tiles 602 or superpixels can begenerated by tile generator 122 that progressively increase from thequality of the low quality tile to the quality of the high quality tile.In some embodiments, the transitional tiles 602 progressively increasein quality to facilitate ensuring that the user does not notice thetransition between render qualities on adjacent tiles 602 with differentrender qualities. For example, the transitional tiles 602 can be used toprovide various levels of progressively increasing or decreasingintermediate image quality between tiles 602 with different qualities.Tile generator 122 can identify tiles 602 that are adjacent each otherwith significantly different image quality (e.g., high image quality andlow image quality) and generate or define the transitional tiles 602 toprovide various levels of image/render quality/resolution therebetween(e.g., several levels of image quality between high image quality andlow image quality). Advantageously, the transitional tiles 602 canreduce aliasing between tiles 602 with different image qualities.

Referring to FIGS. 14-15, another example of the dynamic definition andgeneration of tiles 602 is shown, according to some embodiments. Theexample shown in FIGS. 14-15 is for a different display 164 that has adifferent size and/or resolution as display 164 in FIGS. 12-13,according to some embodiments. In some embodiments, the height h andwidth w of tiles 602 are defined by tile generator 122 independently. Insome embodiments, a proportion or ratio between the height h and width wof tiles 602 is uniform (e.g., all tiles 602 have a h:w ratio of 1:1,all tiles 602 have a h:w ratio of 1:1.618, etc.). In some embodiments,for a same display 164 as the display 164 shown in FIGS. 14-15, a sizeof tiles 602 is the same or has a same area. In this way, higherresolution tiles can still hold a same amount of samples/pixels toprovide high quality imagery. For lower resolution tiles, pixels may beskipped horizontally or vertically.

In response to gaze location 402 moving from the position shown in FIG.14 to the position shown in FIG. 15 (e.g., near the intersection betweenfour tiles 602), tile generator 122 defines a new arrangement, layout,size, etc., of tiles 602, shown as new tiles 602 n, according to someembodiments. New tiles 602 n are shown in dashed lines, with previouslyused tiles 602 shown in solid lines in FIG. 15, according to someembodiments. It should be noted that tile generator 122 defines newtiles 602 n such that gaze location 402 is at a centerpoint of acorresponding new tile 602 n, according to some embodiments. In someembodiments, tile generator 122 defines new tiles 602 n such that error604 is completely within a single new tile 602 n that gaze location 402is within. In some embodiments, tile generator 122 first defines the newtile 602 n centered about the location/position of gaze location 402. Insome embodiments, tile generator 122 proceeds to generating, defining,etc., additional tiles to fill out display 164 in response to definingthe first new tile 602 n that is centered about the location/position ofgaze location 402. Tile generator 164 first defines the layout,arrangement, sizes, boundaries, intersections, etc., of new tiles 602 n,then proceeds to assign rendering resolutions to each of new tiles 602n, according to some embodiments. In some embodiments, tile generator164 provides the definitions of new tiles 602 n and corresponding renderresolutions of each of new tiles 602 n to image renderer 124.

In some embodiments, tile generator 122 temporarily decreases orincreases the sizes of tiles 602 prior to updating/redefining tiles 602(e.g., prior to processing circuitry 116 using new tile 602 n as opposedto the old tiles 602). For example, tile generator 122 can temporarilyadjust (e.g., increase or decrease) the height h and/or width w of tiles602 near or at gaze location 402 before new tile 602 n is used toprovide imagery to the user, according to some embodiments.Advantageously, this reduces the likelihood that the user will noticethe change in tiles 602 (e.g., the repositioning, redefinition,resizing, etc., of tiles 602) and facilitates a seamless transition fromthe old tiles 602 to new tiles 602 n.

It should be understood that tile generator 122 can dynamically updateany of the arrangement, mapping, size, dimensions, correspondingresolutions, etc., of any of tiles 602, according to some embodiments.In some embodiments, tile generator 122 dynamically updates any of thearrangement, mapping, size, dimensions, corresponding resolutions, etc.,of any of tiles 602 independently. In some embodiments, tile generator122 dynamically updates or redefines tile 602 in response to changes inthe location/position of gaze location 402 (or gaze vector 302), changesin the user's gaze, changes in gaze vector 302, changes in a focalpoint, changes in error 604, changes in errors associated with angles θ₁and θ₂, changes in errors associated with gaze vector 302 (and/or gazelocation 402), etc. In some embodiments, tile generator 122 dynamicallyupdates or redefines tiles 602 prior to rendering and displaying a newframe of imagery on display 164.

Tiling Based on Lens Regions

Referring now to FIG. 21, a front view of a curved lens 2102 of display164 is shown, according to some embodiments. Lens 2102 can includevarious quality regions, according to some embodiments. In someembodiments, the quality of regions 2104-2108 may vary based on thedirection of gaze vector 302 (or the location of gaze location 402) ordue to manufacturing and image displaying capabilities of lens 2102and/or display 164. In some embodiments, regions 2104-2108 of lens 2102have different values of a modulation transfer function (MTF). MTFvalues range from 1 to 0, with an MTF value of 1 indicating perfectcontrast preservation, and lower MTF values indicating deterioratedcontrast preservation. In addition, off-axis MTF sagittal and tangentialcomponents may have different values. For example, sagittal component ofMTF could be worse than tangential in a Fresnel lens. In this case, afoveated tile can be made smaller in a sagittal direction and larger ina tangential direction to improve performance.

For example, center region 2108 may have an MTF value between 0.7 and 1,medial regions 2106 may have MTF values between 0.7 and 0.4, and outerregions 2104 of lens 2102 may have MTF values less than 0.4. In someembodiments, areas or regions of lens 2102 that are closer to the centerof lens 2102 have higher MTF values, with regions or areas that arefarther from the center of lens 2102 have lower MTF values.

For example, if certain regions of lens 2102 are viewed at a moretangential angle, the image quality may decrease due to the tangentialviewing angle (the lower MTF values), according to some embodiments.FIG. 21 represents the case when the user is looking directly aheadtowards a center of lens 2102, according to some embodiments. In thiscase, a center region 2108 of lens 2102 is being viewed at a moreperpendicular angle than regions 2106 and regions 2104, according tosome embodiments. Center region 2108 of lens 2102 may have the highestquality due to the viewing angle (e.g., the most perpendicularly viewedarea) and image displaying capabilities of lens 2102.

Tile generator 122 can identify various quality regions of lens 2102 andadjust the size, height, width, number of tiles 602, etc., for thevarious regions of lens 2102, according to some embodiments. In someembodiments, the height h and width w of tiles 602 are generated/definedby tile generator 122 based on an MTF value of a corresponding region orarea of lens 2102. In some embodiments, more tangential or outer regions2104 of lens 2102 (e.g., a left most region, a right most region, anupper most region, a lower most region, etc.) have lower quality (e.g.,lower MTF values), and therefore tile generator 122 determines thatlarger tiles 602 a should be defined/generated for outer regions 2104(or tiles 602 with a lower image quality). Likewise, tile generator 122can identify that central region 2108 has the best capability fordisplaying high resolution/high detail/high quality images (e.g., thehighest MTF values), according to some embodiments. Tile generator 122can generate smallest tiles 602 c for central region 2108, since centralregion 2108 is capable of being viewed the most perpendicularly (and hasthe highest MTF values), according to some embodiments. Medial regions2106 are between central region 2108 and outer regions 2104 and can havemedium sized tiles 602 b to provide medium quality imagery to the user,according to some embodiments.

In general, the height h of tiles 602 can be directly proportional to acorresponding MTF value at the particular location of lens 2102 that thetile 602 will be displayed:h∝MTFaccording to some embodiments. Likewise, the width w of tiles 602 can bedirectly proportional to the corresponding MTF value at the particularlocation of lens 2102 that the tile 602 will be displayed:w∝MTFaccording to some embodiments. The area A of each tile 602 is directlyproportional to the corresponding MTF value:A∝MTFaccording to some embodiments.

It should be understood that tile generator 122 can identify any numberof various regions that are capable of being viewed perpendicularly(e.g., define different regions based on MTF values), according to someembodiments. The ability to view a region more perpendicularly ispositively related to image quality, detail, resolution, etc., that canbe viewed in this region, according to some embodiments. Tile generator122 can identify regions based on how perpendicularly the region can beviewed by the user, and adjust the size of tiles 602 such that regionswhich are capable of being viewed more perpendicularly are associatedwith higher quality imagery or smaller tiles 602, according to someembodiments. In this way, tile generator 122 may adjust the size oftiles 602 and/or the quality of images displayed on the tiles 602 toaccount for image displaying capabilities of lens 2102, according tosome embodiments. Advantageously, this reduces processing power bydisplaying lower quality images in regions which can only be viewed froma tangential angle and are unable to provide high quality images due tothe viewing angle, according to some embodiments.

Dynamic Tiling Process

Referring particularly to FIG. 16, a process 1600 for dynamically tilingand defining tiles is shown, according to some embodiments. Process 1600includes steps 1602-1618 and is performed by system 100, system 200,etc., to provide foveated rendering with dynamically updated tiling,according to some embodiments. In some embodiments, process 1600facilitates reducing the processing power of processing circuitry thatperforms process 1600. Advantageously, process 1600 can be performed bymobile display systems (e.g., head mounted displays, head wearabledisplays, etc.) that do not have adequate processing power to render anentire image in high resolution. In some embodiments, steps 1602-1612are performed concurrently with steps 1614-1618.

Process 1600 includes obtaining gaze direction data (step 1602),according to some embodiments. In some embodiments, the gaze directiondata includes a position of a user's eyes, a position of the user'spupils, an orientation of the user's eyes, etc. In some embodiments, thegaze direction data indicates a direction of the user's gaze. In someembodiments, step 1602 is performed by one or more of sensors 104. Insome embodiments, the gaze direction data includes head direction data.In some embodiments, the gaze direction data includes both eyedirection/position data and head direction/position data. In someembodiments, the gaze direction or eye tracking data is provided to eyetracker 118. In some embodiments, the gaze direction data is gaze vector136, gaze vector 302, gaze location 402, and/or eye tracking data 148.In some embodiments, step 1602 includes obtaining eye tracking data(e.g., eye tracking data 148 and/or head tracking data) and determininga gaze vector (e.g., gaze vector 302) based on the eye tracking data asthe gaze direction data. In some embodiments, the gaze direction dataincludes angles θ₁ and angle θ₂. In some embodiments, step 1602 is atleast partially performed by sensors 104 and/or eye tracker 118.

Process 1600 includes determining a location/position on a display basedon the gaze direction data (step 1604), according to some embodiments.In some embodiments, step 1604 includes determining a localposition/location of a point of interest, a focal point, a gaze point,etc., that the user is looking at or directing their gaze towards on thedisplay (e.g., display 164). In some embodiments, the local positionincludes an x position and a y position (e.g., Cartesian coordinates) ondisplay 164 relative to an origin, a center of display 164, a referencepoint, a corner of display 164, etc. In some embodiments, thelocation/position on the display is a point that is of interest to theuser. In some embodiments, step 1604 is performed by tile generator 122and/or eye tracker 118. In some embodiments, the location/positiondetermined in step 1604 is the position/location of gaze location 402.

Process 1600 includes obtaining errors associated with the gazedirection data and/or the location/position on the display (step 1606),according to some embodiments. In some embodiments, the errors obtainedin step 1606 are errors associated with gaze vector 302, gaze vector136, eye tracking data 148, head tracking data 150, thelocation/position of gaze location 402, angles θ₁ and θ₂ of gaze vector302, etc. In some embodiments, step 1606 is performed by error manager120 based on any of eye tracking data 148, gaze vector 136, headtracking data 150, etc.

Process 1600 includes determining if the location/position on thedisplay is off-center of a corresponding tile or is near a tile border(step 1608), according to some embodiments. In some embodiments, step1608 includes comparing the location/position on the display (e.g., thelocation/position of gaze location 402) to a correspondinglocation/position of tile 602 that the location/position is within. Insome embodiments, the location/position on the display deviating fromthe location/position of the center of the corresponding tile indicatesthat the user is directing their gaze towards a different location ondisplay 164. In some embodiments, step 1608 includes determining if thelocation/position on the display (e.g., the location/position of gazelocation 402) is near a border of an adjacent tile. In some embodiments,step 1608 includes checking within an area of the display defined by theerror to determine if an adjacent border or intersection is within thearea. In some embodiments, step 1608 is optional. In some embodiments,step 1608 is performed by tile generator 122.

Process 1600 includes defining new tiles based on the location/positionon the display (step 1610), according to some embodiments. In someembodiments, step 1610 is performed by tile generator 122. In someembodiments, a first new tile is defined that is centered about thelocation/position on the display that the user is directing their gazetowards. In some embodiments, additional tiles are generated thatsurround the first new tile and fill-out the rest of the display. Insome embodiments, a size of the tiles (e.g., a height and a width) ispredetermined. In some embodiments, the ratio between the height and thewidth of the new tiles is a predetermined or fixed ratio. In someembodiments, the size, shape, area, etc., of the new tiles aredetermined or defined based on the error determined in step 1606. Insome embodiments, the size, shape, area, etc., of the new tiles arepredetermined.

Process 1600 includes assigning various render resolutions to each ofthe tiles for foveated rendering (step 1612), according to someembodiments. In some embodiments, step 1612 is performed by tilegenerator 122. In some embodiments, various render qualities are used.For example, some tiles can be assigned a high resolution renderquality, other tiles may be assigned a medium resolution render quality,other tiles a low resolution render quality, etc. In some embodiments,any number of different render qualities are used. For example, therender qualities can include a high resolution and a low resolutionquality, according to some embodiments. In some embodiments, the renderresolutions are assigned to each tile (e.g., tiles 602) to achievefoveated rendering. For example, tiles at or near the location/positionthat the user is directing their gaze or focusing upon can be assignedhigh resolution render quality, according to some embodiments. In someembodiments, tiles that are farther away from the location/position onthe display that the user is focusing upon have lower render quality. Insome embodiments, tiles directly adjacent the first new tile have amedium or high resolution quality. In some embodiments, tiles other thanthe high and medium resolution quality tiles are assigned the lowresolution render quality.

Process 1600 includes receiving image data (step 1614), according tosome embodiments. In some embodiments, the image data is image data 132.In some embodiments, the image data includes three dimensional objects,characters, particles, pre-rendered images, textures, materials, lights,etc. In some embodiments, step 1614 is performed by image renderer 124.

Process 1600 includes rendering portions of the image data correspondingto each of the new tiles (e.g., the tiles defined in step 1610)according to the render resolutions (e.g., as assigned or defined instep 1612) of each new tile (step 1616), according to some embodiments.In some embodiments, step 1616 is performed by a render engine. In someembodiments, step 1616 is performed by image renderer 124. In someembodiments, rendering an image includes rendering or rasterizing theimage at a render quality and upscaling the rendered image to fit acorresponding tile. For example, portions of the image data that will bedisplayed on low-render quality tiles may be rendered or rasterized atlow quality, then up-scaled to fit the size of the tiles, according tosome embodiments. Likewise, portions of the image that will be displayedin medium-quality tiles can be rendered or rasterized at medium quality,then up-scaled to fit the size of the tiles, according to someembodiments.

Process 1600 includes providing the rendered image (e.g., the imagerendered in step 1616) to a user on the display (step 1618), accordingto some embodiments. In some embodiments, the various portions of theimage that are rendered or rasterized in step 1616 are assembled anddisplayed on corresponding tiles of the display and provided to theuser. In some embodiments, step 1618 is performed by image renderer 124and/or display(s) 164.

Process 1600 can return to step 1602 in response to performing step1618, according to some embodiments. In some embodiments, process 1600is performed prior to displaying a new frame to the user on the display.In some embodiments, process 1600 facilitates dynamically re-updatingand/or re-defining the tiles to provide reduced processing powerfoveated rendering. Advantageously, process 1600 allows low processingpower display systems to track the user's eyes, determine a focal point,and render foveated images to the user based on the focal point of theuser's eyes, according to some embodiments.

Referring now to FIG. 17, a process 1700 for defining or generatingtiles for a tiled foveated rendering system is shown, according to someembodiments. In some embodiments, process 1700 is performed byprocessing circuitry 116. In some embodiments, process 1700 is performedto define new tiles and to assign render resolutions to the new tiles(steps 1610 and 1612).

Process 1700 includes obtaining gaze position (step 1702), according tosome embodiments. In some embodiments, the gaze position is thelocation/position of gaze location 402 on display 164. In someembodiments, the gaze position is or includes angles θ₁ and θ₂ (shown inFIGS. 3-5). In some embodiments, step 1702 is the same as or similar tosteps 1602 and 1604 of process 1600. In some embodiments, step 1702 isperformed by eye tracker 118 based on eye tracking data 148. In someembodiments, the gaze position (e.g., gaze location 402) is a point ondisplay 164 that gaze vector 136/302 is directed towards.

Process 1700 includes determining width and height of one or more tilesbased on errors associated with gaze angles and/or based on opticalquality regions of a lens (step 1704), according to some embodiments. Insome embodiments, higher errors associated with the gaze angles (e.g.,angles θ₁ and/or θ₂) correspond to larger values of the width w andheight h of one or more tiles. In some embodiments, the error associatedwith angle θ₁ is associated with the height h of the one or more tiles.In some embodiments, the error associated with angle θ₂ is associatedwith the width w of the one or more tiles. In some embodiments, step1704 is performed by tile generator 122. In some embodiments, the errorsassociated with the gaze angles are determined by error manager 120based on eye tracking data 148. In some embodiments, other errors areused to determine the width w and the height h of the one or more tiles.For example, errors in the x and y position of the gaze position ondisplay 164 can be used to determine the height h and the width w of theone or more tiles, according to some embodiments. In some embodiments,the gaze position is the location/position of gaze location 402. In someembodiments, the errors are any of errors associated with angles θ₁ andθ₂, the x and y positions of gaze location 402, etc.

In some embodiments, step 1704 includes determining a range oflocations, angles, etc., where the gaze position may be. For example,the error can be used to determine a range of values of angle θ₁ suchas: θ_(1,range)=θ₁±Δθ₁ and: θ_(2,range)=θ₂±Δθ₂, where θ_(1,range) is arange of values of the angle θ₁, Δθ₁ is the error associated with θ₁,θ_(2,range) is a range of values of the angle θ₂, and Δθ₂ is the errorassociated with θ₂. In some embodiments, the range of values for anglesθ₁ and θ₂ translates to a range of locations/positions on display 164.For example, θ_(1,range) can translate to a range of y positions Δy ondisplay 164, and θ_(2,range) can translate to a range of x positions Δxon display 164, according to some embodiments. In some embodiments, theranges of x locations/positions and y locations/positions is representedin FIGS. 6-15 as error 604. Likewise, the errors associated with anglesθ₁ and θ₂ can translates to errors associated with the x position andthe y position of the gaze position on display 164.

Step 1704 can include using any of the errors associated with the gazedirection (e.g., associated with gaze vector 302), the gaze position(e.g., errors associated with the position/location of gaze location402) to determine or choose the height h and the width w of the one ormore tiles, according to some embodiments. In some embodiments, theerrors are used to independently select, determine, choose, etc., theheight h and width w of the one or more tiles (e.g., the errorassociated with the x position of gaze location 402 is used to determinethe width w of the one or more tiles, and the error associated with they position of gaze location 402 is used to determine the height h of theone or more tiles). In some embodiments, the height h and the width w ofthe one or more tiles is a predetermined or predefined ratio.

In some embodiments, regional quality of a lens of the display is usedto determine the shape, size, etc., of the one or more tiles. In someembodiments, for example, a lens of display 164 can have lower qualityregions (e.g., around a perimeter or at an outer periphery of the lens)and higher quality regions (e.g., at or near a center of the lens). Insome embodiments, the height h and width w of the one or more tiles arechosen, selected, determined, etc., based on the quality of a region ofthe lens at which each of the one or more tiles will be displayed. Forexample, if a tile will be displayed at a lower quality region of thelens of the display, the tile may have a smaller height h and width w,according to some embodiments. Likewise, if a tile will be displayed ata higher quality region of the lens of the display, the tile may have alarger height h and width w, according to some embodiments. In someembodiments, a combination of both the regional quality of the lens andthe errors associated with the gaze angles (or the gaze position) areused to determine the height h and the width w of each of the one ormore tiles. In some embodiments, each of the one or more tiles has auniform height h and width w. In some embodiments, the one or more tileshave a non-uniform height h and width w. For example a tile centered atthe gaze position may have a higher value of the height h and width w ifthe gaze position is at a higher resolution region or area of the lensof the display. Likewise, a tile centered at the gaze position may havea lower value of the height h and width w if the gaze position is at alower resolution/quality region.

In some embodiments, the values of the height h and the width w of eachof the one or more tiles is proportional or directly related to theresolution quality of a corresponding region of the lens of the displaywhere the tiles are positioned (e.g., higher resolution quality areas ofthe lens of the display correspond to larger tiles). In someembodiments, the values of the height h and the width w of each of theone or more tiles are inversely proportional or inversely related to theresolution quality of the corresponding region of the lens of thedisplay where the tiles are position (e.g., lower resolution qualityareas/regions of the lens of the display correspond to smaller tiles).

Process 1700 includes generating a tile with the determined width andheight at the obtained gaze position (step 1706), according to someembodiments. In some embodiments, the generated tile is positioned atthe obtained gaze position. In some embodiments, the generated tile hasthe width w and height h as determined in step 1704 based on the errorsassociated with the gaze angles and/or the optical quality region of thelens of the display. In some embodiments, the tile is defined andgenerated by tile generator 122. In some embodiments, the tile is arender region with high quality.

Process 1700 includes generating additional tiles to fill-out a displayarea of the display (step 1708), according to some embodiments. In someembodiments, additional tiles are generated to fill out remaining areaof the display that the tile generated in step 1706 does not cover. Theadditional tiles have a height h and a width w that can be determined instep 1704 and/or can be determined using any of the techniques describedin greater detail above with reference to step 1704 (e.g., based onerrors associated with gaze angles, based on regional quality of a lensof the display, etc.), according to some embodiments. In someembodiments, the additional tiles have non-uniform height h and width w.In some embodiment, step 1708 is performed by tile generator 122.

Process 1700 includes assigning render qualities to each of the tiles,with the highest rendering quality centered about the gaze position(step 1710), according to some embodiments. In some embodiments,centering the highest resolution/rendering quality at the gaze positionfacilitates foveated rendering of imagery on the display. In someembodiments, step 1710 is performed by tile generator 122. In someembodiments, various predetermined quality levels are assigned to eachof the tiles. For example, tile generator 122 can assign any of a highquality, a medium quality, a lower quality, etc., to each of the tiles,according to some embodiments. In some embodiments, tile generator 122uses more or less than three predetermined quality levels for each ofthe tiles. In some embodiments, the render quality of each of the tilesis determined based on a distance of each tile from the tile that iscentered about the gaze position. In some embodiments, the renderquality assigned to tiles decreases with increased distance from thetile that is centered at the gaze position. In this way, tiles that areviewed in the user's peripheral vision may have reducedresolution/quality, thereby facilitating improved processingcapabilities, according to some embodiments.

In some embodiments, process 1700 proceeds to step 1614 of process 1600in response to completing step 1710. In some embodiments, process 1700redefines the tiles for subsequent iterations of process 1700. In someembodiments, if the gaze position is still sufficiently at or near thecenter of a previously generated high resolution quality tile, thepreviously generated tiles are maintained. For example, if a usermaintains their gaze near or at the center of a high resolution qualitytile that is previously generated, the previously generated tiles can bemaintained, according to some embodiments. In some embodiments, if thegaze position (e.g., the position/location of gaze location 402 ondisplay 164) changes significantly (e.g., shifts towards another tile,shifts to a lower resolution quality tile, etc.), process 1700determines that tiles should be re-generated/redefined to account forthe change in the gaze position. In some embodiments, process 1700includes a step or process of temporarily decreasing the size of thetiles (e.g., the height h and/or width w) prior to displaying imagerywith the newly defined tiles. In some embodiments, process 1700 includestemporarily increasing the resolution quality of tiles during atransition to new tile definition/layout. In this way, transitionsbetween new tile definitions, layouts, sizes, positions, etc., may besmoothed to facilitate reducing the likelihood that the user will noticethe transition, according to some embodiments.

Referring now to FIG. 18, a process 1800 for determining if the tilesshould be redefined or re-generated is shown, according to someembodiments. Process 1800 includes steps 1802-1808, according to someembodiments. In some embodiments, process 1800 is performed by tilegenerator 122. In some embodiments, process 1800 is performed prior toperforming process 1700. Process 1800 is performed by tile generator 122to determine if the tiles (e.g., tiles 602) of the display (e.g.,display 164) should be redefined, resized, rearranged, etc., accordingto some embodiments.

Process 1800 includes determining a difference between the gaze positionand a center of a corresponding tile (step 1802), according to someembodiments. In some embodiments, step 1802 includes determining adifference in an x direction and a y direction between the center of thecorresponding tile and the gaze position. In some embodiments, step 1802includes determining a difference between an angular value of angle θ₁that corresponds to the center of the corresponding tile and an angularvalue of angle θ₁ that corresponds to the gaze position. In someembodiments, step 1802 includes determining a difference between anangular value of angle θ₂ that corresponds to the center of thecorresponding tile and an angular value of angle θ₂ that corresponds tothe gaze position. The corresponding tile is the tile that gaze positionis currently or was previously centered at (or centered near), accordingto some embodiments.

Process 1800 includes determining if the difference as determined instep 1802 is greater than (or greater than or equal to) a thresholdvalue (step 1804), according to some embodiments. In some embodiments,step 1804 includes comparing any of the deviation of the x and/or yvalues of the gaze position on the display from the centerpoint of thecorresponding tile to one or more threshold values. In some embodiments,step 1804 includes comparing a magnitude of deviation between the gazeposition on the display and the centerpoint of the corresponding tile toa threshold value. The threshold value is proportional to at least oneof the height h and the width w of the corresponding tile, according tosome embodiments. For example, the threshold value can be 10% of thewidth w of the corresponding tile, 25% of the height h of thecorresponding tile, etc., according to some embodiments. In someembodiments, step 1804 is performed by tile generator 122.

Process 1800 includes maintaining a current tiling arrangement ordefinition (step 1806) in response to the difference being less than thethreshold value(s) (step 1804, “NO”), according to some embodiments. Insome embodiments, if the gaze position does not deviate from the centerpoint of the corresponding tile by a predetermined amount (e.g., thethreshold value in step 1804), process 1800 proceeds to step 1806 andmaintains current tiling. In some embodiments, step 1806 is performed bytile generator 122.

Process 1800 includes updating the tiling by performing process 1700 inresponse to the difference being greater than (or greater than or equalto) the threshold value (step 1808), according to some embodiments. Insome embodiments, if the difference is greater than (or greater than orequal to) the threshold value (step 1804, “YES”), process 1800 proceedsto step 1808. In some embodiments, step 1808 is performed by tilegenerator 122. In some embodiments, step 1808 includes performingprocess 1700 and various steps of process 1600. In some embodiments,process 1800 includes an additional step of displaying one or moreframes with reduced tile size prior to performing process 1700, or priorto performing step 1618 of process 1600 to facilitate a smoothtransition between current or previous tiling arrangement/definitionsand subsequent/updated tiling arrangements/definitions.

Referring now to FIG. 19, a process 1900 for determining if the tilesshould be redefined or re-generated is shown, according to someembodiments. Process 1900 includes steps 1902-1904, and steps 1806 and1808, according to some embodiments. In some embodiments, process 1900is similar to process 1800. In some embodiments, process 1900 iscombined with process 1800. In some embodiments, process 1900 isperformed by tile generator 122.

Process 1900 includes determining a range of possible gaze positionsbased on errors associated with gaze angles (step 1902), according tosome embodiments. In some embodiments, step 1902 includes determining arange of locations, positions, etc., on display 164 that the user's gazemay be directed (e.g., error 604) based on errors associated with theuser's gaze angles (e.g., based on gaze error 126, errors associatedwith angles θ₁ and θ₂, etc.). In some embodiments, the range of possiblegaze positions is a region or area of display 164. In some embodiments,the range of possible gaze positions is centered about the gaze position(e.g., centered about gaze location 402). In some embodiments, step 1902is performed by error manager 120 and/or tile generator 122.

Process 1900 includes determining if a tile border is within the rangeof possible gaze positions (step 1904), according to some embodiments.In some embodiments, the tile border is any of a border between adjacenttiles, an intersection between one or more tiles, an intersectionbetween a corner of one or more tiles, etc. In response to a tile borderbeing within the range of possible gaze positions (step 1904, “YES”),process 1900 proceeds to step 1808, according to some embodiments. Inresponse to a tile border not being within the range of possible gazepositions (step 1904, “NO”), process 1900 proceeds to step 1806. In someembodiments, step 1904 is performed by tile generator 122.

In some embodiments, process 1900 and 1800 are combined. For example,step 1904 and all subsequent steps of process 1900 can be performed inresponse to step 1804 of process 1800 (e.g., in response to “NO” of step1804, or in response to “YES” of step 1804), according to someembodiments. In some embodiments, step 1804 and all subsequent steps ofprocess 1800 are performed in response to step 1904 of process 1900(e.g., in response to “YES” of step 1904, or in response to “NO” of step1904).

Server System

Various operations described herein can be implemented on computersystems. FIG. 20 shows a block diagram of a representative server system2000 and client computer system 2014 usable to implement the presentdisclosure. Server system 2000 or similar systems can implement servicesor servers described herein or portions thereof. Client computer system2014 or similar systems can implement clients described herein. Each ofsystems 100, 200 and others described herein can incorporate features ofsystems 2000, 2014.

Server system 2000 can have a modular design that incorporates a numberof modules 2002 (e.g., blades in a blade server); while two modules 2002are shown, any number can be provided. Each module 2002 can includeprocessing unit(s) 2004 and local storage 2006.

Processing unit(s) 2004 can include a single processor, which can haveone or more cores, or multiple processors. Processing unit(s) 2004 caninclude a general-purpose primary processor as well as one or morespecial-purpose co-processors such as graphics processors, digitalsignal processors, or the like. Some or all processing units 2004 can beimplemented using customized circuits, such as application specificintegrated circuits (ASICs) or field programmable gate arrays (FPGAs).Such integrated circuits execute instructions that are stored on thecircuit itself. Processing unit(s) 2004 can execute instructions storedin local storage 2006. Any type of processors in any combination can beincluded in processing unit(s) 2004.

Local storage 2006 can include volatile storage media (e.g.,conventional DRAM, SRAM, SDRAM, or the like) and/or non-volatile storagemedia (e.g., magnetic or optical disk, flash memory, or the like).Storage media incorporated in local storage 2006 can be fixed, removableor upgradeable as desired. Local storage 2006 can be physically orlogically divided into various subunits such as a system memory, aread-only memory (ROM), and a permanent storage device. The systemmemory can be a read-and-write memory device or a volatileread-and-write memory, such as dynamic random-access memory. The systemmemory can store some or all of the instructions and data thatprocessing unit(s) 2004 need at runtime. The ROM can store static dataand instructions that are needed by processing unit(s) 2004. Thepermanent storage device can be a non-volatile read-and-write memorydevice that can store instructions and data even when module 2002 ispowered down. The term “storage medium” as used herein includes anymedium in which data can be stored indefinitely (subject to overwriting,electrical disturbance, power loss, or the like) and does not includecarrier waves and transitory electronic signals propagating wirelesslyor over wired connections.

Local storage 2006 can store one or more software programs to beexecuted by processing unit(s) 2004, such as an operating system and/orprograms implementing various server functions such as functions of thesystem 100, or any other system described herein, or any other server(s)associated with the system 100 or any other system described herein.

“Software” refers generally to sequences of instructions that, whenexecuted by processing unit(s) 2004 cause server system 2000 (orportions thereof) to perform various operations, thus defining one ormore specific machine implementations that execute and perform theoperations of the software programs. The instructions can be stored asfirmware residing in read-only memory and/or program code stored innon-volatile storage media that can be read into volatile working memoryfor execution by processing unit(s) 2004. Software can be implemented asa single program or a collection of separate programs or program modulesthat interact as desired. From local storage 2006 (or non-local storagedescribed below), processing unit(s) 2004 can retrieve programinstructions to execute and data to process in order to execute variousoperations described above.

In some server systems 2000, multiple modules 2002 can be interconnectedvia a bus or other interconnect 2008, forming a local area network thatsupports communication between modules 2002 and other components ofserver system 2000. Interconnect 2008 can be implemented using varioustechnologies including server racks, hubs, routers, etc.

A wide area network (WAN) interface 2010 can provide data communicationcapability between the local area network (interconnect 2008) and alarger network, such as the Internet. Conventional or other activitiestechnologies can be used, including wired (e.g., Ethernet, IEEE 802.3standards) and/or wireless technologies (e.g., Wi-Fi, IEEE 802.11standards).

Local storage 2006 can provide working memory for processing unit(s)2004, providing fast access to programs and/or data to be processedwhile reducing traffic on interconnect 2008. Storage for largerquantities of data can be provided on the local area network by one ormore mass storage subsystems 2012 that can be connected to interconnect2008. Mass storage subsystem 2012 can be based on magnetic, optical,semiconductor, or other data storage media. Direct attached storage,storage area networks, network-attached storage, and the like can beused. Any data stores or other collections of data described herein asbeing produced, consumed, or maintained by a service or server can bestored in mass storage subsystem 2012. Additional data storage resourcesmay be accessible via WAN interface 2010 (potentially with increasedlatency).

Server system 2000 can operate in response to requests received via WANinterface 2010. For example, one of modules 2002 can implement asupervisory function and assign discrete tasks to other modules 2002 inresponse to received requests. Conventional work allocation techniquescan be used. As requests are processed, results can be returned to therequester via WAN interface 2010. Such operation can generally beautomated. WAN interface 2010 can connect multiple server systems 2000to each other, providing scalable systems capable of managing highvolumes of activity. Conventional or other techniques for managingserver systems and server farms (collections of server systems thatcooperate) can be used, including dynamic resource allocation andreallocation.

Server system 2000 can interact with various user-owned or user-operateddevices via a wide-area network such as the Internet. An example of auser-operated device is shown in FIG. 20 as client computing system2014. Client computing system 2014 can be implemented, for example, as aconsumer device such as a smartphone, other mobile phone, tabletcomputer, wearable computing device (e.g., smart watch, eyeglasses),desktop computer, laptop computer, and so on.

For example, client computing system 2014 can communicate via WANinterface 2010. Client computing system 2014 can include conventionalcomputer components such as processing unit(s) 2016, storage device2018, network interface 2020, user input device 2022, and user outputdevice 2024. Client computing system 2014 can be a computing deviceimplemented in a variety of form factors, such as a desktop computer,laptop computer, tablet computer, smartphone, other mobile computingdevice, wearable computing device, or the like.

Processor 2016 and storage device 2018 can be similar to processingunit(s) 2004 and local storage 2006 described above. Suitable devicescan be selected based on the demands to be placed on client computingsystem 2014; for example, client computing system 2014 can beimplemented as a “thin” client with limited processing capability or asa high-powered computing device. Client computing system 2014 can beprovisioned with program code executable by processing unit(s) 2016 toenable various interactions with server system 2000 of a messagemanagement service such as accessing messages, performing actions onmessages, and other interactions described above. Some client computingsystems 2014 can also interact with a messaging service independently ofthe message management service.

Network interface 2020 can provide a connection to a wide area network(e.g., the Internet) to which WAN interface 2010 of server system 2000is also connected. Network interface 2020 can include a wired interface(e.g., Ethernet) and/or a wireless interface implementing various RFdata communication standards such as Wi-Fi, Bluetooth, or cellular datanetwork standards (e.g., 3G, 4G, LTE, etc.).

User input device 2022 can include any device (or devices) via which auser can provide signals to client computing system 2014; clientcomputing system 2014 can interpret the signals as indicative ofparticular user requests or information. User input device 2022 caninclude any or all of a keyboard, touch pad, touch screen, mouse orother pointing device, scroll wheel, click wheel, dial, button, switch,keypad, microphone, and so on.

User output device 2024 can include any device via which clientcomputing system 2014 can provide information to a user. For example,user output device 2024 can include a display to display imagesgenerated by or delivered to client computing system 2014. The displaycan incorporate various image generation technologies, e.g., a liquidcrystal display (LCD), light-emitting diode (LED) including organiclight-emitting diodes (OLED), projection system, cathode ray tube (CRT),or the like, together with supporting electronics (e.g.,digital-to-analog or analog-to-digital converters, signal processors, orthe like). A device such as a touchscreen that function as both inputand output device can be used. Output devices 2024 can be provided inaddition to or instead of a display. Examples include indicator lights,speakers, tactile “display” devices, printers, and so on.

Configuration of Exemplary Embodiments

Having now described some illustrative implementations, it is apparentthat the foregoing is illustrative and not limiting, having beenpresented by way of example. In particular, although many of theexamples presented herein involve specific combinations of method actsor system elements, those acts and those elements can be combined inother ways to accomplish the same objectives. Acts, elements andfeatures discussed in connection with one implementation are notintended to be excluded from a similar role in other implementations orimplementations.

The hardware and data processing components used to implement thevarious processes, operations, illustrative logics, logical blocks,modules and circuits described in connection with the embodimentsdisclosed herein may be implemented or performed with a general purposesingle- or multi-chip processor, a digital signal processor (DSP), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), or other programmable logic device, discrete gate ortransistor logic, discrete hardware components, or any combinationthereof designed to perform the functions described herein. A generalpurpose processor may be a microprocessor, or, any conventionalprocessor, controller, microcontroller, or state machine. A processoralso may be implemented as a combination of computing devices, such as acombination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. In some embodiments, particularprocesses and methods may be performed by circuitry that is specific toa given function. The memory (e.g., memory, memory unit, storage device,etc.) may include one or more devices (e.g., RAM, ROM, Flash memory,hard disk storage, etc.) for storing data and/or computer code forcompleting or facilitating the various processes, layers and modulesdescribed in the present disclosure. The memory may be or includevolatile memory or non-volatile memory, and may include databasecomponents, object code components, script components, or any other typeof information structure for supporting the various activities andinformation structures described in the present disclosure. According toan exemplary embodiment, the memory is communicably connected to theprocessor via a processing circuit and includes computer code forexecuting (e.g., by the processing circuit and/or the processor) the oneor more processes described herein.

The present disclosure contemplates methods, systems and programproducts on any machine-readable media for accomplishing variousoperations. The embodiments of the present disclosure may be implementedusing existing computer processors, or by a special purpose computerprocessor for an appropriate system, incorporated for this or anotherpurpose, or by a hardwired system. Embodiments within the scope of thepresent disclosure include program products comprising machine-readablemedia for carrying or having machine-executable instructions or datastructures stored thereon. Such machine-readable media can be anyavailable media that can be accessed by a general purpose or specialpurpose computer or other machine with a processor. By way of example,such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, orother optical disk storage, magnetic disk storage or other magneticstorage devices, or any other medium which can be used to carry or storedesired program code in the form of machine-executable instructions ordata structures and which can be accessed by a general purpose orspecial purpose computer or other machine with a processor. Combinationsof the above are also included within the scope of machine-readablemedia. Machine-executable instructions include, for example,instructions and data which cause a general purpose computer, specialpurpose computer, or special purpose processing machines to perform acertain function or group of functions.

The phraseology and terminology used herein is for the purpose ofdescription and should not be regarded as limiting. The use of“including” “comprising” “having” “containing” “involving”“characterized by” “characterized in that” and variations thereofherein, is meant to encompass the items listed thereafter, equivalentsthereof, and additional items, as well as alternate implementationsconsisting of the items listed thereafter exclusively. In oneimplementation, the systems and methods described herein consist of one,each combination of more than one, or all of the described elements,acts, or components.

Any references to implementations or elements or acts of the systems andmethods herein referred to in the singular can also embraceimplementations including a plurality of these elements, and anyreferences in plural to any implementation or element or act herein canalso embrace implementations including only a single element. Referencesin the singular or plural form are not intended to limit the presentlydisclosed systems or methods, their components, acts, or elements tosingle or plural configurations. References to any act or element beingbased on any information, act or element can include implementationswhere the act or element is based at least in part on any information,act, or element.

Any implementation disclosed herein can be combined with any otherimplementation or embodiment, and references to “an implementation,”“some implementations,” “one implementation” or the like are notnecessarily mutually exclusive and are intended to indicate that aparticular feature, structure, or characteristic described in connectionwith the implementation can be included in at least one implementationor embodiment. Such terms as used herein are not necessarily allreferring to the same implementation. Any implementation can be combinedwith any other implementation, inclusively or exclusively, in any mannerconsistent with the aspects and implementations disclosed herein.

Where technical features in the drawings, detailed description or anyclaim are followed by reference signs, the reference signs have beenincluded to increase the intelligibility of the drawings, detaileddescription, and claims. Accordingly, neither the reference signs northeir absence have any limiting effect on the scope of any claimelements.

Systems and methods described herein may be embodied in other specificforms without departing from the characteristics thereof. Furtherrelative parallel, perpendicular, vertical or other positioning ororientation descriptions include variations within +/−10% or +/−10degrees of pure vertical, parallel or perpendicular positioning.References to “approximately,” “about” “substantially” or other terms ofdegree include variations of +/−10% from the given measurement, unit, orrange unless explicitly indicated otherwise. Coupled elements can beelectrically, mechanically, or physically coupled with one anotherdirectly or with intervening elements. Scope of the systems and methodsdescribed herein is thus indicated by the appended claims, rather thanthe foregoing description, and changes that come within the meaning andrange of equivalency of the claims are embraced therein.

The term “coupled” and variations thereof includes the joining of twomembers directly or indirectly to one another. Such joining may bestationary (e.g., permanent or fixed) or moveable (e.g., removable orreleasable). Such joining may be achieved with the two members coupleddirectly with or to each other, with the two members coupled with eachother using a separate intervening member and any additionalintermediate members coupled with one another, or with the two memberscoupled with each other using an intervening member that is integrallyformed as a single unitary body with one of the two members. If“coupled” or variations thereof are modified by an additional term(e.g., directly coupled), the generic definition of “coupled” providedabove is modified by the plain language meaning of the additional term(e.g., “directly coupled” means the joining of two members without anyseparate intervening member), resulting in a narrower definition thanthe generic definition of “coupled” provided above. Such coupling may bemechanical, electrical, or fluidic.

References to “or” can be construed as inclusive so that any termsdescribed using “or” can indicate any of a single, more than one, andall of the described terms. A reference to “at least one of ‘A’ and B”can include only ‘A’, only ‘B’, as well as both ‘A’ and ‘B’. Suchreferences used in conjunction with “comprising” or other openterminology can include additional items.

Modifications of described elements and acts such as variations insizes, dimensions, structures, shapes and proportions of the variouselements, values of parameters, mounting arrangements, use of materials,colors, orientations can occur without materially departing from theteachings and advantages of the subject matter disclosed herein. Forexample, elements shown as integrally formed can be constructed ofmultiple parts or elements, the position of elements can be reversed orotherwise varied, and the nature or number of discrete elements orpositions can be altered or varied. Other substitutions, modifications,changes and omissions can also be made in the design, operatingconditions and arrangement of the disclosed elements and operationswithout departing from the scope of the present disclosure.

References herein to the positions of elements (e.g., “top,” “bottom,”“above,” “below”) are merely used to describe the orientation of variouselements in the FIGURES. The orientation of various elements may differaccording to other exemplary embodiments, and that such variations areintended to be encompassed by the present disclosure.

What is claimed is:
 1. A method of providing imagery to a user on adisplay, the method comprising: receiving eye tracking data; determininga gaze location on the display or a gaze vector using the eye trackingdata; defining a first tile using the gaze location on the display orthe gaze vector, wherein the first tile has a height and a width, theheight and width being determined using the eye tracking data; defininga plurality of additional tiles to fill an entire area of the display;and providing a portion of an image using the first tile at a firstimage quality and providing another portion of the image at a secondimage quality using at least one of the plurality of additional tiles.2. The method of claim 1, wherein the eye tracking data indicates adirection of gaze of the user.
 3. The method of claim 2, whereinobtaining eye tracking data comprises monitoring an angular orientationof the user's eye about a first axis and a second axis to determine thedirection of gaze of the user.
 4. The method of claim 1, wherein theheight or the width of the first tile is defined based on any of, or acombination of: a vertical or horizontal error of the gaze location; arate of change of the vertical or horizontal error of the gaze location;a vertical or horizontal position of the gaze location; an MTF value ofa lens of the display at the gaze location; and a rate of change of thevertical or horizontal position of the gaze location.
 5. The method ofclaim 1, further comprising: re-defining the first tile and theplurality of additional tiles in response to a change of the gazelocation to a new gaze location on the display; wherein the first tileis redefined to be centered at the new gaze location.
 6. The method ofclaim 5, wherein the first tile and the plurality of additional tilesare redefined in response to the gaze location on the display changingat least a threshold amount.
 7. The method of claim 5, wherein the firsttile and the plurality of additional tiles are redefined in response tothe direction of gaze changing at least a threshold angular amount in avertical direction or a horizontal direction.
 8. The method of claim 1,further comprising: determining an error associated with the directionof gaze of the user; defining at least one of the height and the widthof the first tile based on the error associated with the direction ofgaze of the user.
 9. The method of claim 1, further comprising: definingone or more regions related to image displaying capabilities ofdifferent areas of the display; and defining a size of the first andeach of the plurality of additional tiles based on which of the one ormore regions the tiles are displayed.
 10. The method of claim 9, whereinthe one or more regions are defined based on a magnitude of a tangentialviewing angle for the different areas of the display.
 11. A head mounteddisplay for providing imagery to a user, the head mounted displaycomprising: a display; an eye tracker configured to provide eye trackingdata; and a processor configured to: determine a gaze location on thedisplay or a gaze vector using the eye tracking data; define a firsttile using the gaze location or the gaze vector; and define a pluralityof additional tiles to fill an area of the display; wherein a portion ofan image is provided on the display using the first tile at a firstimage quality and other portions of the image are provided on thedisplay at a second image quality using at least one of the plurality ofadditional tiles.
 12. The head mounted display of claim 11, wherein theprocessor is configured to monitor an angular orientation of the user'seye about a first axis and a second axis to determine a direction ofgaze of the user.
 13. The head mounted display of claim 11, wherein thesecond image quality is lower than the first image quality.
 14. The headmounted display of claim 11, wherein the processor is configured todefine a height or a width of the first tile based on any of, or acombination of: a vertical error of the gaze location; a rate of changeof the vertical error of the gaze location; a vertical position of thegaze location; a modulation transfer function value of a lens of thedisplay at the gaze location; and a rate of change of the verticalposition of the gaze location.
 15. The head mounted display of claim 11,wherein the processing circuitry is further configured to: redefine thefirst tile and the plurality of additional tiles in response to a changeof the gaze location to a new gaze location on the display; wherein thefirst tile is redefined to be centered at the new gaze location.
 16. Thehead mounted display of claim 11, wherein the processor is configured todetermine an error associated with the direction of gaze of the user anddefine at least one of a height and a width of the first tile based onthe error associated with the direction of gaze of the user.
 17. Thehead mounted display of claim 11, wherein the processor is configuredto: identify one or more regions based on image displaying capabilitiesof different areas of the display; and define a size of the first andeach of the plurality of additional tiles based on which of the one ormore regions the tiles are displayed.
 18. The head mounted display ofclaim 17, wherein the processor is configured to identify the one ormore regions based on a magnitude of a tangential viewing angle for thedifferent areas of the display.
 19. A display for providing foveatedimagery to a user, the display comprising processing circuitryconfigured to: track a gaze direction or a gaze vector of the user'seye; define a first tile based on the gaze direction or gaze vector ofthe user's eye, wherein a location of the first tile is defined usingthe gaze direction or the gaze vector and is for a first image quality;define one or more additional tiles for a second image quality differentthan the first image quality; provide imagery using the first tile andeach of the one or more additional tiles; and redefine the location ofthe first tile in response to a change in the gaze direction or gazevector.
 20. The display of claim 19, wherein the processing circuitry isconfigured to repeat the steps of defining the first tile, defining theone or more additional tiles, and providing the imagery in response toat least one of: the gaze direction of the user's eye changing at leasta threshold amount; and the gaze direction of the user's eye beingwithin a threshold distance from a border between the first tile and anadjacent tile.